Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 20:08:39.471827 29217 test_util.cc:268] Using random seed: 225235621
I20240501 20:08:39.483323 29217 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:39.483413 29217 ts_itest-base.cc:113] --------------
I20240501 20:08:39.483457 29217 ts_itest-base.cc:114] 3 tablet servers
I20240501 20:08:39.483525 29217 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:39.483585 29217 ts_itest-base.cc:116] --------------
2024-05-01T20:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:39Z Disabled control of system clock
I20240501 20:08:39.506407 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:36805
--webserver_interface=127.28.136.126
--webserver_port=0
--builtin_ntp_servers=127.28.136.84:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.136.126:36805
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:08:39.834225 29229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:39.834537 29229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:39.834656 29229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:39.842680 29229 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:39.842819 29229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:39.842900 29229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:39.842972 29229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:39.843040 29229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:39.852373 29229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.136.126:36805
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.136.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:39.853729 29229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:39.855518 29229 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:39.866174 29234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:39.866576 29235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:39.867105 29237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:39.867695 29229 server_base.cc:1034] running on GCE node
I20240501 20:08:39.868476 29229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:39.870595 29229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:39.871788 29229 hybrid_clock.cc:648] HybridClock initialized: now 1714594119871737 us; error 45 us; skew 500 ppm
I20240501 20:08:39.872104 29229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:39.878108 29229 webserver.cc:460] Webserver started at http://127.28.136.126:35551/ using document root <none> and password file <none>
I20240501 20:08:39.878923 29229 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:39.879043 29229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:39.879356 29229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:39.882045 29229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d"
format_stamp: "Formatted at 2024-05-01 20:08:39 on dist-test-slave-sx0n"
I20240501 20:08:39.882783 29229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d"
format_stamp: "Formatted at 2024-05-01 20:08:39 on dist-test-slave-sx0n"
I20240501 20:08:39.888240 29229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:08:39.891705 29243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:39.893090 29229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:39.893244 29229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d"
format_stamp: "Formatted at 2024-05-01 20:08:39 on dist-test-slave-sx0n"
I20240501 20:08:39.893430 29229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:39.908332 29229 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:39.909165 29229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:39.909425 29229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:39.926266 29229 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.126:36805
I20240501 20:08:39.926285 29294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.126:36805 every 8 connection(s)
I20240501 20:08:39.927597 29229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:39.929601 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 29229
I20240501 20:08:39.929832 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:39.931473 29295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:39.942143 29295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d: Bootstrap starting.
I20240501 20:08:39.946453 29295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:39.947897 29295 log.cc:824] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:39.951166 29295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d: No bootstrap required, opened a new log
I20240501 20:08:39.958644 29295 raft_consensus.cc:348] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36805 } }
I20240501 20:08:39.958987 29295 raft_consensus.cc:374] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:39.959096 29295 raft_consensus.cc:729] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33569ff8d5c4d2b84bcff7d06fc6f7d, State: Initialized, Role: FOLLOWER
I20240501 20:08:39.959697 29295 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36805 } }
I20240501 20:08:39.959918 29295 raft_consensus.cc:388] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:39.960048 29295 raft_consensus.cc:482] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:39.960192 29295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:39.980077 29295 raft_consensus.cc:504] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36805 } }
I20240501 20:08:39.980844 29295 leader_election.cc:302] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d33569ff8d5c4d2b84bcff7d06fc6f7d; no voters:
I20240501 20:08:39.981582 29295 leader_election.cc:288] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:39.981698 29300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:39.982263 29300 raft_consensus.cc:686] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [term 1 LEADER]: Becoming Leader. State: Replica: d33569ff8d5c4d2b84bcff7d06fc6f7d, State: Running, Role: LEADER
I20240501 20:08:39.982915 29300 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36805 } }
I20240501 20:08:39.985634 29295 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:39.985922 29301 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33569ff8d5c4d2b84bcff7d06fc6f7d. Latest consensus state: current_term: 1 leader_uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36805 } } }
I20240501 20:08:39.986244 29301 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:39.986302 29302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33569ff8d5c4d2b84bcff7d06fc6f7d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36805 } } }
I20240501 20:08:39.986532 29302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:39.989683 29304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:39.995863 29304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:40.004798 29304 catalog_manager.cc:1347] Generated new cluster ID: 97a47b6041824c229423e026b59754f1
I20240501 20:08:40.004922 29304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:40.025455 29304 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:40.026767 29304 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:40.035722 29304 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d33569ff8d5c4d2b84bcff7d06fc6f7d: Generated new TSK 0
I20240501 20:08:40.036801 29304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:40.052714 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.65:0
--local_ip_for_outbound_sockets=127.28.136.65
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36805
--builtin_ntp_servers=127.28.136.84:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:08:40.384541 29319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:40.384861 29319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:40.384991 29319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:40.392465 29319 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:40.392585 29319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:40.392760 29319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.65
I20240501 20:08:40.403995 29319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.65
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:40.405333 29319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:40.407061 29319 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:40.418370 29324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:40.418771 29325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:40.419317 29327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:40.419816 29319 server_base.cc:1034] running on GCE node
I20240501 20:08:40.420349 29319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:40.421247 29319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:40.422435 29319 hybrid_clock.cc:648] HybridClock initialized: now 1714594120422390 us; error 40 us; skew 500 ppm
I20240501 20:08:40.422724 29319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:40.424795 29319 webserver.cc:460] Webserver started at http://127.28.136.65:41441/ using document root <none> and password file <none>
I20240501 20:08:40.425503 29319 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:40.425611 29319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:40.425907 29319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:40.428427 29319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "44483838ca544377b883694ab69979fe"
format_stamp: "Formatted at 2024-05-01 20:08:40 on dist-test-slave-sx0n"
I20240501 20:08:40.429174 29319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "44483838ca544377b883694ab69979fe"
format_stamp: "Formatted at 2024-05-01 20:08:40 on dist-test-slave-sx0n"
I20240501 20:08:40.434031 29319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:40.437274 29333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.438620 29319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:40.438774 29319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "44483838ca544377b883694ab69979fe"
format_stamp: "Formatted at 2024-05-01 20:08:40 on dist-test-slave-sx0n"
I20240501 20:08:40.438957 29319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:40.450840 29319 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:40.451596 29319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:40.451843 29319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:40.452679 29319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:40.454063 29319 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:40.454154 29319 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.454252 29319 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:40.454316 29319 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.483876 29319 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.65:45509
I20240501 20:08:40.483922 29445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.65:45509 every 8 connection(s)
I20240501 20:08:40.485198 29319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:40.490087 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 29319
I20240501 20:08:40.490278 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:40.493770 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.66:0
--local_ip_for_outbound_sockets=127.28.136.66
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36805
--builtin_ntp_servers=127.28.136.84:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:40.498549 29446 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36805
I20240501 20:08:40.498840 29446 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:40.499640 29446 heartbeater.cc:511] Master 127.28.136.126:36805 requested a full tablet report, sending...
I20240501 20:08:40.501597 29260 ts_manager.cc:194] Registered new tserver with Master: 44483838ca544377b883694ab69979fe (127.28.136.65:45509)
I20240501 20:08:40.503065 29260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.65:41189
W20240501 20:08:40.828632 29450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:40.828977 29450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:40.829079 29450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:40.836416 29450 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:40.836529 29450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:40.836647 29450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.66
I20240501 20:08:40.845671 29450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.66
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:40.846999 29450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:40.848668 29450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:40.861050 29455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:40.861547 29456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:40.862109 29458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:40.862638 29450 server_base.cc:1034] running on GCE node
I20240501 20:08:40.863286 29450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:40.864235 29450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:40.865422 29450 hybrid_clock.cc:648] HybridClock initialized: now 1714594120865368 us; error 50 us; skew 500 ppm
I20240501 20:08:40.865731 29450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:40.867849 29450 webserver.cc:460] Webserver started at http://127.28.136.66:43537/ using document root <none> and password file <none>
I20240501 20:08:40.868569 29450 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:40.868678 29450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:40.868990 29450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:40.871534 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "62f50c608e55425abb9d25821764451c"
format_stamp: "Formatted at 2024-05-01 20:08:40 on dist-test-slave-sx0n"
I20240501 20:08:40.872264 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "62f50c608e55425abb9d25821764451c"
format_stamp: "Formatted at 2024-05-01 20:08:40 on dist-test-slave-sx0n"
I20240501 20:08:40.877362 29450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:08:40.881032 29464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.882349 29450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:40.882505 29450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "62f50c608e55425abb9d25821764451c"
format_stamp: "Formatted at 2024-05-01 20:08:40 on dist-test-slave-sx0n"
I20240501 20:08:40.882691 29450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:40.915549 29450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:40.916553 29450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:40.916916 29450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:40.918110 29450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:40.920102 29450 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:40.920218 29450 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.920347 29450 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:40.920426 29450 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:40.952718 29450 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.66:38189
I20240501 20:08:40.952792 29576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.66:38189 every 8 connection(s)
I20240501 20:08:40.954264 29450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:40.955154 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 29450
I20240501 20:08:40.955341 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:40.959236 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:0
--local_ip_for_outbound_sockets=127.28.136.67
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36805
--builtin_ntp_servers=127.28.136.84:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:40.967830 29577 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36805
I20240501 20:08:40.968178 29577 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:40.969170 29577 heartbeater.cc:511] Master 127.28.136.126:36805 requested a full tablet report, sending...
I20240501 20:08:40.970783 29260 ts_manager.cc:194] Registered new tserver with Master: 62f50c608e55425abb9d25821764451c (127.28.136.66:38189)
I20240501 20:08:40.971612 29260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.66:58277
W20240501 20:08:41.277251 29580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:41.277562 29580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:41.277701 29580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:41.285149 29580 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:41.285276 29580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:41.285431 29580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:08:41.294682 29580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:41.295955 29580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:41.297698 29580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:41.309705 29586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:41.310220 29585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:41.310542 29588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:41.311133 29580 server_base.cc:1034] running on GCE node
I20240501 20:08:41.311712 29580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:41.312644 29580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:41.313846 29580 hybrid_clock.cc:648] HybridClock initialized: now 1714594121313785 us; error 56 us; skew 500 ppm
I20240501 20:08:41.314221 29580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:41.320456 29580 webserver.cc:460] Webserver started at http://127.28.136.67:37957/ using document root <none> and password file <none>
I20240501 20:08:41.321327 29580 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:41.321453 29580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:41.321784 29580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:41.324451 29580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "17b77e4a12564b67ab2849497540a431"
format_stamp: "Formatted at 2024-05-01 20:08:41 on dist-test-slave-sx0n"
I20240501 20:08:41.325292 29580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "17b77e4a12564b67ab2849497540a431"
format_stamp: "Formatted at 2024-05-01 20:08:41 on dist-test-slave-sx0n"
I20240501 20:08:41.330834 29580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:41.334483 29594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:41.335947 29580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:41.336115 29580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "17b77e4a12564b67ab2849497540a431"
format_stamp: "Formatted at 2024-05-01 20:08:41 on dist-test-slave-sx0n"
I20240501 20:08:41.336316 29580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:41.354961 29580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:41.355892 29580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:41.356171 29580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:41.357235 29580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:41.358803 29580 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:41.358891 29580 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:41.358991 29580 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:41.359050 29580 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:41.389212 29580 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:45141
I20240501 20:08:41.389258 29706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:45141 every 8 connection(s)
I20240501 20:08:41.390483 29580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:41.396415 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 29580
I20240501 20:08:41.396600 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:41.403066 29707 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36805
I20240501 20:08:41.403353 29707 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:41.404155 29707 heartbeater.cc:511] Master 127.28.136.126:36805 requested a full tablet report, sending...
I20240501 20:08:41.405479 29260 ts_manager.cc:194] Registered new tserver with Master: 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:41.406395 29260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:33159
I20240501 20:08:41.412559 29217 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:08:41.433080 29217 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 20:08:41.505867 29446 heartbeater.cc:503] Master 127.28.136.126:36805 was elected leader, sending a full tablet report...
I20240501 20:08:41.974861 29577 heartbeater.cc:503] Master 127.28.136.126:36805 was elected leader, sending a full tablet report...
I20240501 20:08:42.408658 29707 heartbeater.cc:503] Master 127.28.136.126:36805 was elected leader, sending a full tablet report...
I20240501 20:08:45.187336 29260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52480:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240501 20:08:45.193673 29260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:08:45.234145 29375 tablet_service.cc:1467] Processing CreateTablet for tablet 7be4b9ce1050425eb8483d182ad0fb01 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.233724 29379 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba4d209d9a14ff594d4014b38d589b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.236202 29379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba4d209d9a14ff594d4014b38d589b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.236120 29374 tablet_service.cc:1467] Processing CreateTablet for tablet 03911cb24ebc476a943bb7bd06878319 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.236665 29374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03911cb24ebc476a943bb7bd06878319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.238327 29373 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0a8b308c8c46188e04cbc0d7605b34 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.238862 29373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0a8b308c8c46188e04cbc0d7605b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.237941 29640 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba4d209d9a14ff594d4014b38d589b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.239912 29372 tablet_service.cc:1467] Processing CreateTablet for tablet bd6fffc18bfc40bcb0c2dbb4a87d5851 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.240479 29372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6fffc18bfc40bcb0c2dbb4a87d5851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.240911 29639 tablet_service.cc:1467] Processing CreateTablet for tablet c06cf612821e4a1492c72f501e254a1a (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.241910 29635 tablet_service.cc:1467] Processing CreateTablet for tablet 03911cb24ebc476a943bb7bd06878319 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.242373 29371 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8b4b93b0d341279ffabb59631d3b84 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.242872 29371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8b4b93b0d341279ffabb59631d3b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.243044 29633 tablet_service.cc:1467] Processing CreateTablet for tablet bd6fffc18bfc40bcb0c2dbb4a87d5851 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.243775 29641 tablet_service.cc:1467] Processing CreateTablet for tablet 40dac71d185243e481b67806a29f7717 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.243688 29505 tablet_service.cc:1467] Processing CreateTablet for tablet 03911cb24ebc476a943bb7bd06878319 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.245170 29370 tablet_service.cc:1467] Processing CreateTablet for tablet ddf0d60db6f845b7ac0e1bc99ddfa4c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.245642 29370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf0d60db6f845b7ac0e1bc99ddfa4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.245669 29505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03911cb24ebc476a943bb7bd06878319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.245925 29631 tablet_service.cc:1467] Processing CreateTablet for tablet ddf0d60db6f845b7ac0e1bc99ddfa4c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.246003 29636 tablet_service.cc:1467] Processing CreateTablet for tablet 7be4b9ce1050425eb8483d182ad0fb01 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.246881 29637 tablet_service.cc:1467] Processing CreateTablet for tablet 195b8d506a36424b8c734e9b998ff7cb (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.246954 29638 tablet_service.cc:1467] Processing CreateTablet for tablet c98260be534042778599a404180811a1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.246990 29632 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8b4b93b0d341279ffabb59631d3b84 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.246133 29634 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0a8b308c8c46188e04cbc0d7605b34 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.248672 29642 tablet_service.cc:1467] Processing CreateTablet for tablet 0765346a16a14413bbeab82888bbc194 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.249184 29503 tablet_service.cc:1467] Processing CreateTablet for tablet bd6fffc18bfc40bcb0c2dbb4a87d5851 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.249207 29369 tablet_service.cc:1467] Processing CreateTablet for tablet 59a41ea289d74d43a0218dafb4d4778c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.249641 29369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a41ea289d74d43a0218dafb4d4778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.249661 29503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6fffc18bfc40bcb0c2dbb4a87d5851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.250049 29628 tablet_service.cc:1467] Processing CreateTablet for tablet 2a92ab6a8ce24735a223363676c1bea6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.250203 29630 tablet_service.cc:1467] Processing CreateTablet for tablet 59a41ea289d74d43a0218dafb4d4778c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.250547 29501 tablet_service.cc:1467] Processing CreateTablet for tablet ddf0d60db6f845b7ac0e1bc99ddfa4c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.251003 29501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf0d60db6f845b7ac0e1bc99ddfa4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.251084 29629 tablet_service.cc:1467] Processing CreateTablet for tablet 87c6dcd194e148e3b8446a571a73eb69 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.251823 29499 tablet_service.cc:1467] Processing CreateTablet for tablet 87c6dcd194e148e3b8446a571a73eb69 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.251823 29627 tablet_service.cc:1467] Processing CreateTablet for tablet 844da4163c3a440cbc20bae219ea7edd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.252244 29499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c6dcd194e148e3b8446a571a73eb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.252776 29497 tablet_service.cc:1467] Processing CreateTablet for tablet 844da4163c3a440cbc20bae219ea7edd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.253248 29497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844da4163c3a440cbc20bae219ea7edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.253224 29368 tablet_service.cc:1467] Processing CreateTablet for tablet 87c6dcd194e148e3b8446a571a73eb69 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.253547 29640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba4d209d9a14ff594d4014b38d589b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.253564 29507 tablet_service.cc:1467] Processing CreateTablet for tablet 195b8d506a36424b8c734e9b998ff7cb (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.253763 29368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c6dcd194e148e3b8446a571a73eb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.253948 29507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195b8d506a36424b8c734e9b998ff7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.254007 29628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a92ab6a8ce24735a223363676c1bea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.254127 29509 tablet_service.cc:1467] Processing CreateTablet for tablet c06cf612821e4a1492c72f501e254a1a (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.254464 29632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8b4b93b0d341279ffabb59631d3b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.254513 29509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06cf612821e4a1492c72f501e254a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.254992 29627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844da4163c3a440cbc20bae219ea7edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.256148 29511 tablet_service.cc:1467] Processing CreateTablet for tablet 40dac71d185243e481b67806a29f7717 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.256501 29638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98260be534042778599a404180811a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.256541 29511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40dac71d185243e481b67806a29f7717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.256907 29631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddf0d60db6f845b7ac0e1bc99ddfa4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.257179 29367 tablet_service.cc:1467] Processing CreateTablet for tablet 2a92ab6a8ce24735a223363676c1bea6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.257304 29629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c6dcd194e148e3b8446a571a73eb69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.257611 29367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a92ab6a8ce24735a223363676c1bea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.257726 29633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6fffc18bfc40bcb0c2dbb4a87d5851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.258157 29634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0a8b308c8c46188e04cbc0d7605b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.258550 29636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be4b9ce1050425eb8483d182ad0fb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.258942 29637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195b8d506a36424b8c734e9b998ff7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.259352 29639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06cf612821e4a1492c72f501e254a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.259749 29630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a41ea289d74d43a0218dafb4d4778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.260139 29635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03911cb24ebc476a943bb7bd06878319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.261170 29366 tablet_service.cc:1467] Processing CreateTablet for tablet 844da4163c3a440cbc20bae219ea7edd (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.261584 29366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 844da4163c3a440cbc20bae219ea7edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.265064 29512 tablet_service.cc:1467] Processing CreateTablet for tablet 0765346a16a14413bbeab82888bbc194 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.265585 29512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0765346a16a14413bbeab82888bbc194. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.269308 29510 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba4d209d9a14ff594d4014b38d589b4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.269975 29510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba4d209d9a14ff594d4014b38d589b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.273332 29508 tablet_service.cc:1467] Processing CreateTablet for tablet c98260be534042778599a404180811a1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.273895 29508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98260be534042778599a404180811a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.277232 29506 tablet_service.cc:1467] Processing CreateTablet for tablet 7be4b9ce1050425eb8483d182ad0fb01 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.277791 29506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be4b9ce1050425eb8483d182ad0fb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.279457 29726 tablet_bootstrap.cc:492] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.282023 29504 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0a8b308c8c46188e04cbc0d7605b34 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.282032 29377 tablet_service.cc:1467] Processing CreateTablet for tablet c98260be534042778599a404180811a1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.282521 29504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0a8b308c8c46188e04cbc0d7605b34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.282646 29377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c98260be534042778599a404180811a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.284806 29726 tablet_bootstrap.cc:654] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.285198 29502 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8b4b93b0d341279ffabb59631d3b84 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.285728 29502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8b4b93b0d341279ffabb59631d3b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.286032 29378 tablet_service.cc:1467] Processing CreateTablet for tablet c06cf612821e4a1492c72f501e254a1a (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.286412 29726 log.cc:824] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:45.286502 29378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06cf612821e4a1492c72f501e254a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.289018 29380 tablet_service.cc:1467] Processing CreateTablet for tablet 40dac71d185243e481b67806a29f7717 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.289274 29500 tablet_service.cc:1467] Processing CreateTablet for tablet 59a41ea289d74d43a0218dafb4d4778c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.289552 29380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40dac71d185243e481b67806a29f7717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.289757 29500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a41ea289d74d43a0218dafb4d4778c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.292948 29381 tablet_service.cc:1467] Processing CreateTablet for tablet 0765346a16a14413bbeab82888bbc194 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.293042 29498 tablet_service.cc:1467] Processing CreateTablet for tablet 2a92ab6a8ce24735a223363676c1bea6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240501 20:08:45.293437 29381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0765346a16a14413bbeab82888bbc194. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.293506 29498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a92ab6a8ce24735a223363676c1bea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.299559 29376 tablet_service.cc:1467] Processing CreateTablet for tablet 195b8d506a36424b8c734e9b998ff7cb (DEFAULT_TABLE table=rebalance_test_table_0 [id=c2dbf7765bdd4d10b3ae837d83a916f8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240501 20:08:45.300053 29376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195b8d506a36424b8c734e9b998ff7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.300921 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40dac71d185243e481b67806a29f7717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.304924 29642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0765346a16a14413bbeab82888bbc194. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.306596 29375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be4b9ce1050425eb8483d182ad0fb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.360555 29728 tablet_bootstrap.cc:492] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.366223 29728 tablet_bootstrap.cc:654] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.367126 29726 tablet_bootstrap.cc:492] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.367333 29728 log.cc:824] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:45.367805 29726 ts_tablet_manager.cc:1367] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.088s user 0.025s sys 0.058s
I20240501 20:08:45.369091 29725 tablet_bootstrap.cc:492] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.370846 29728 tablet_bootstrap.cc:492] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.371129 29728 ts_tablet_manager.cc:1367] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240501 20:08:45.372115 29725 tablet_bootstrap.cc:654] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.373453 29725 log.cc:824] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:45.375999 29725 tablet_bootstrap.cc:492] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.376248 29725 ts_tablet_manager.cc:1367] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 20:08:45.376384 29726 raft_consensus.cc:348] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.376829 29726 raft_consensus.cc:374] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.376993 29726 raft_consensus.cc:729] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.377705 29726 consensus_queue.cc:260] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.379487 29726 ts_tablet_manager.cc:1398] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20240501 20:08:45.379475 29728 raft_consensus.cc:348] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.379943 29726 tablet_bootstrap.cc:492] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.379947 29728 raft_consensus.cc:374] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.380048 29728 raft_consensus.cc:729] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.380831 29728 consensus_queue.cc:260] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.382189 29726 tablet_bootstrap.cc:654] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.384654 29725 raft_consensus.cc:348] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.385118 29725 raft_consensus.cc:374] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.385236 29725 raft_consensus.cc:729] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.385960 29725 consensus_queue.cc:260] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.388831 29728 ts_tablet_manager.cc:1398] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.018s user 0.007s sys 0.012s
I20240501 20:08:45.389355 29728 tablet_bootstrap.cc:492] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.390518 29726 tablet_bootstrap.cc:492] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.390679 29726 ts_tablet_manager.cc:1367] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240501 20:08:45.391073 29728 tablet_bootstrap.cc:654] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.391587 29725 ts_tablet_manager.cc:1398] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240501 20:08:45.391669 29726 raft_consensus.cc:348] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.391880 29726 raft_consensus.cc:374] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.391990 29726 raft_consensus.cc:729] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.392292 29726 consensus_queue.cc:260] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.392834 29726 ts_tablet_manager.cc:1398] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:08:45.393141 29726 tablet_bootstrap.cc:492] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
W20240501 20:08:45.394342 29708 tablet.cc:2369] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:45.394953 29726 tablet_bootstrap.cc:654] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.400087 29726 tablet_bootstrap.cc:492] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.400211 29725 tablet_bootstrap.cc:492] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.400261 29726 ts_tablet_manager.cc:1367] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:08:45.401012 29726 raft_consensus.cc:348] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.401275 29726 raft_consensus.cc:374] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.401360 29726 raft_consensus.cc:729] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.401712 29726 consensus_queue.cc:260] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.402057 29725 tablet_bootstrap.cc:654] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.403650 29726 ts_tablet_manager.cc:1398] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:45.403961 29726 tablet_bootstrap.cc:492] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.404151 29728 tablet_bootstrap.cc:492] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.404300 29728 ts_tablet_manager.cc:1367] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20240501 20:08:45.405303 29728 raft_consensus.cc:348] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.405519 29728 raft_consensus.cc:374] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.405588 29726 tablet_bootstrap.cc:654] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.405614 29728 raft_consensus.cc:729] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.405880 29728 consensus_queue.cc:260] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.406410 29728 ts_tablet_manager.cc:1398] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:08:45.406708 29728 tablet_bootstrap.cc:492] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.408390 29728 tablet_bootstrap.cc:654] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.410760 29725 tablet_bootstrap.cc:492] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.410943 29725 ts_tablet_manager.cc:1367] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:08:45.411782 29725 raft_consensus.cc:348] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.411973 29725 raft_consensus.cc:374] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.412068 29725 raft_consensus.cc:729] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.412312 29725 consensus_queue.cc:260] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.412986 29725 ts_tablet_manager.cc:1398] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:45.413269 29725 tablet_bootstrap.cc:492] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.415045 29725 tablet_bootstrap.cc:654] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.416548 29728 tablet_bootstrap.cc:492] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.417085 29728 ts_tablet_manager.cc:1367] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240501 20:08:45.417897 29728 raft_consensus.cc:348] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.418139 29728 raft_consensus.cc:374] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.418212 29728 raft_consensus.cc:729] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.418468 29728 consensus_queue.cc:260] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.418864 29725 tablet_bootstrap.cc:492] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.418942 29728 ts_tablet_manager.cc:1398] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.419034 29725 ts_tablet_manager.cc:1367] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240501 20:08:45.419279 29728 tablet_bootstrap.cc:492] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.419751 29725 raft_consensus.cc:348] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.420032 29725 raft_consensus.cc:374] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.420128 29725 raft_consensus.cc:729] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.420408 29725 consensus_queue.cc:260] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.420611 29728 tablet_bootstrap.cc:654] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.421166 29725 ts_tablet_manager.cc:1398] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:08:45.421478 29725 tablet_bootstrap.cc:492] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.423167 29725 tablet_bootstrap.cc:654] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.425082 29726 tablet_bootstrap.cc:492] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.425257 29726 ts_tablet_manager.cc:1367] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.006s
I20240501 20:08:45.426084 29726 raft_consensus.cc:348] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.426303 29726 raft_consensus.cc:374] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.426398 29726 raft_consensus.cc:729] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.426646 29728 tablet_bootstrap.cc:492] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.426614 29726 consensus_queue.cc:260] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.426780 29728 ts_tablet_manager.cc:1367] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240501 20:08:45.427150 29726 ts_tablet_manager.cc:1398] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:08:45.427425 29726 tablet_bootstrap.cc:492] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.427459 29728 raft_consensus.cc:348] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.427646 29728 raft_consensus.cc:374] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.427733 29728 raft_consensus.cc:729] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.427943 29728 consensus_queue.cc:260] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.428489 29728 ts_tablet_manager.cc:1398] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:08:45.428820 29728 tablet_bootstrap.cc:492] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.429028 29726 tablet_bootstrap.cc:654] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.430362 29728 tablet_bootstrap.cc:654] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.431859 29725 tablet_bootstrap.cc:492] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.432018 29725 ts_tablet_manager.cc:1367] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20240501 20:08:45.432829 29725 raft_consensus.cc:348] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.433043 29725 raft_consensus.cc:374] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.433120 29725 raft_consensus.cc:729] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.433328 29725 consensus_queue.cc:260] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.434501 29725 ts_tablet_manager.cc:1398] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:45.434754 29725 tablet_bootstrap.cc:492] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.436520 29725 tablet_bootstrap.cc:654] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.438431 29726 tablet_bootstrap.cc:492] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.438587 29726 ts_tablet_manager.cc:1367] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.008s
I20240501 20:08:45.438838 29728 tablet_bootstrap.cc:492] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.438979 29728 ts_tablet_manager.cc:1367] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20240501 20:08:45.439272 29725 tablet_bootstrap.cc:492] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.439311 29726 raft_consensus.cc:348] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.439430 29725 ts_tablet_manager.cc:1367] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 20:08:45.439525 29726 raft_consensus.cc:374] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.439605 29726 raft_consensus.cc:729] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.439620 29728 raft_consensus.cc:348] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.439832 29728 raft_consensus.cc:374] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.439906 29728 raft_consensus.cc:729] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.439857 29726 consensus_queue.cc:260] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.440141 29728 consensus_queue.cc:260] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.440116 29725 raft_consensus.cc:348] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.440325 29726 ts_tablet_manager.cc:1398] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.440330 29725 raft_consensus.cc:374] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.440402 29725 raft_consensus.cc:729] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.440620 29726 tablet_bootstrap.cc:492] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.440691 29728 ts_tablet_manager.cc:1398] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.440662 29725 consensus_queue.cc:260] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.441012 29728 tablet_bootstrap.cc:492] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.441607 29725 ts_tablet_manager.cc:1398] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:45.441867 29725 tablet_bootstrap.cc:492] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.442225 29726 tablet_bootstrap.cc:654] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.442622 29728 tablet_bootstrap.cc:654] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.443343 29725 tablet_bootstrap.cc:654] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.450809 29728 tablet_bootstrap.cc:492] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.450992 29728 ts_tablet_manager.cc:1367] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:08:45.451797 29728 raft_consensus.cc:348] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.452011 29728 raft_consensus.cc:374] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.452112 29728 raft_consensus.cc:729] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.452414 29728 consensus_queue.cc:260] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.453886 29728 ts_tablet_manager.cc:1398] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:45.454200 29728 tablet_bootstrap.cc:492] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.455924 29728 tablet_bootstrap.cc:654] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.458055 29725 tablet_bootstrap.cc:492] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.458215 29725 ts_tablet_manager.cc:1367] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20240501 20:08:45.458396 29726 tablet_bootstrap.cc:492] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.458565 29726 ts_tablet_manager.cc:1367] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20240501 20:08:45.458669 29728 tablet_bootstrap.cc:492] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.458843 29728 ts_tablet_manager.cc:1367] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
W20240501 20:08:45.459044 29578 tablet.cc:2369] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:45.458971 29725 raft_consensus.cc:348] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:45.459167 29725 raft_consensus.cc:374] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.459255 29725 raft_consensus.cc:729] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.459261 29726 raft_consensus.cc:348] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.459429 29726 raft_consensus.cc:374] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.459520 29726 raft_consensus.cc:729] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.459509 29725 consensus_queue.cc:260] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:45.459564 29728 raft_consensus.cc:348] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.459775 29728 raft_consensus.cc:374] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.459848 29728 raft_consensus.cc:729] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.459762 29726 consensus_queue.cc:260] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.460351 29726 ts_tablet_manager.cc:1398] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.460533 29725 ts_tablet_manager.cc:1398] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:08:45.460590 29728 consensus_queue.cc:260] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.460813 29725 tablet_bootstrap.cc:492] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.461158 29728 ts_tablet_manager.cc:1398] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:45.461469 29728 tablet_bootstrap.cc:492] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.461532 29726 tablet_bootstrap.cc:492] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.463016 29726 tablet_bootstrap.cc:654] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.463029 29725 tablet_bootstrap.cc:654] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.463059 29728 tablet_bootstrap.cc:654] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.471414 29725 tablet_bootstrap.cc:492] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.471568 29725 ts_tablet_manager.cc:1367] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240501 20:08:45.471694 29726 tablet_bootstrap.cc:492] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.471885 29726 ts_tablet_manager.cc:1367] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240501 20:08:45.472096 29728 tablet_bootstrap.cc:492] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.472112 29725 raft_consensus.cc:348] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.472231 29728 ts_tablet_manager.cc:1367] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240501 20:08:45.472272 29725 raft_consensus.cc:374] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.472316 29725 raft_consensus.cc:729] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.472515 29725 consensus_queue.cc:260] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.473001 29725 ts_tablet_manager.cc:1398] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 20:08:45.472939 29728 raft_consensus.cc:348] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.473122 29728 raft_consensus.cc:374] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.473213 29728 raft_consensus.cc:729] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.472992 29726 raft_consensus.cc:348] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.473285 29725 tablet_bootstrap.cc:492] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.473393 29726 raft_consensus.cc:374] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.473523 29726 raft_consensus.cc:729] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.473423 29728 consensus_queue.cc:260] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.473870 29726 consensus_queue.cc:260] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.474517 29726 ts_tablet_manager.cc:1398] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:08:45.475020 29725 tablet_bootstrap.cc:654] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.475167 29728 ts_tablet_manager.cc:1398] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:45.475437 29728 tablet_bootstrap.cc:492] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.476176 29726 tablet_bootstrap.cc:492] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.477373 29728 tablet_bootstrap.cc:654] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.478216 29726 tablet_bootstrap.cc:654] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.482537 29728 tablet_bootstrap.cc:492] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.482692 29728 ts_tablet_manager.cc:1367] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240501 20:08:45.482828 29725 tablet_bootstrap.cc:492] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.482964 29725 ts_tablet_manager.cc:1367] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240501 20:08:45.483228 29726 tablet_bootstrap.cc:492] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.483351 29726 ts_tablet_manager.cc:1367] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240501 20:08:45.483376 29728 raft_consensus.cc:348] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.483611 29728 raft_consensus.cc:374] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.483696 29728 raft_consensus.cc:729] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.483651 29725 raft_consensus.cc:348] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.483831 29725 raft_consensus.cc:374] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.483989 29725 raft_consensus.cc:729] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.483920 29728 consensus_queue.cc:260] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.484210 29725 consensus_queue.cc:260] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.484408 29728 ts_tablet_manager.cc:1398] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:08:45.484706 29728 tablet_bootstrap.cc:492] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.485853 29725 ts_tablet_manager.cc:1398] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:08:45.486212 29728 tablet_bootstrap.cc:654] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.486200 29725 tablet_bootstrap.cc:492] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.488056 29725 tablet_bootstrap.cc:654] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.491845 29728 tablet_bootstrap.cc:492] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.491987 29728 ts_tablet_manager.cc:1367] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 20:08:45.492374 29725 tablet_bootstrap.cc:492] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.492529 29725 ts_tablet_manager.cc:1367] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:08:45.492678 29728 raft_consensus.cc:348] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.492887 29728 raft_consensus.cc:374] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.492983 29728 raft_consensus.cc:729] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.493196 29728 consensus_queue.cc:260] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.493261 29725 raft_consensus.cc:348] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.493479 29725 raft_consensus.cc:374] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.493559 29725 raft_consensus.cc:729] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.493749 29728 ts_tablet_manager.cc:1398] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:45.493820 29725 consensus_queue.cc:260] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.494020 29728 tablet_bootstrap.cc:492] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.495554 29728 tablet_bootstrap.cc:654] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.493326 29726 raft_consensus.cc:348] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.497331 29725 ts_tablet_manager.cc:1398] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:08:45.497503 29726 raft_consensus.cc:374] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.497656 29726 raft_consensus.cc:729] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.497990 29726 consensus_queue.cc:260] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.498610 29726 ts_tablet_manager.cc:1398] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240501 20:08:45.499428 29726 tablet_bootstrap.cc:492] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.500102 29725 tablet_bootstrap.cc:492] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.501557 29726 tablet_bootstrap.cc:654] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.501982 29725 tablet_bootstrap.cc:654] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.502092 29732 raft_consensus.cc:482] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.502331 29732 raft_consensus.cc:504] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
W20240501 20:08:45.503773 29447 tablet.cc:2369] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:45.504314 29732 leader_election.cc:288] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.514408 29728 tablet_bootstrap.cc:492] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.514638 29728 ts_tablet_manager.cc:1367] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20240501 20:08:45.515486 29728 raft_consensus.cc:348] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.515734 29728 raft_consensus.cc:374] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.515810 29728 raft_consensus.cc:729] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.516081 29732 raft_consensus.cc:482] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.516256 29732 raft_consensus.cc:504] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.516999 29732 leader_election.cc:288] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.516065 29728 consensus_queue.cc:260] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.517585 29731 raft_consensus.cc:482] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.517848 29731 raft_consensus.cc:504] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.518930 29728 ts_tablet_manager.cc:1398] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:08:45.519258 29728 tablet_bootstrap.cc:492] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.520090 29731 leader_election.cc:288] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 44483838ca544377b883694ab69979fe (127.28.136.65:45509)
I20240501 20:08:45.521034 29728 tablet_bootstrap.cc:654] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.524210 29734 raft_consensus.cc:482] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.524475 29734 raft_consensus.cc:504] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.526650 29734 leader_election.cc:288] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.534292 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6fffc18bfc40bcb0c2dbb4a87d5851" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.534808 29662 raft_consensus.cc:2454] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:45.535331 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c6dcd194e148e3b8446a571a73eb69" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.535660 29661 raft_consensus.cc:2454] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:45.536365 29468 leader_election.cc:302] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.536846 29468 leader_election.cc:302] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.537218 29732 raft_consensus.cc:2781] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.537333 29732 raft_consensus.cc:482] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.537433 29732 raft_consensus.cc:3037] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.538597 29734 raft_consensus.cc:482] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.538753 29734 raft_consensus.cc:504] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.539323 29734 leader_election.cc:288] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.540669 29732 raft_consensus.cc:504] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.541924 29732 leader_election.cc:288] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.542423 29732 raft_consensus.cc:2781] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.542750 29732 raft_consensus.cc:482] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.542989 29732 raft_consensus.cc:3037] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.544350 29725 tablet_bootstrap.cc:492] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.544520 29725 ts_tablet_manager.cc:1367] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.012s
I20240501 20:08:45.545403 29725 raft_consensus.cc:348] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.545601 29725 raft_consensus.cc:374] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.545686 29725 raft_consensus.cc:729] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.545904 29725 consensus_queue.cc:260] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.547514 29725 ts_tablet_manager.cc:1398] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:08:45.547821 29725 tablet_bootstrap.cc:492] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.548146 29732 raft_consensus.cc:504] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.549575 29725 tablet_bootstrap.cc:654] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.549824 29732 leader_election.cc:288] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.557382 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6fffc18bfc40bcb0c2dbb4a87d5851" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.557626 29661 raft_consensus.cc:3037] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.560268 29725 tablet_bootstrap.cc:492] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.560422 29725 ts_tablet_manager.cc:1367] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240501 20:08:45.561024 29661 raft_consensus.cc:2454] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:45.561317 29725 raft_consensus.cc:348] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.561547 29725 raft_consensus.cc:374] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.561625 29725 raft_consensus.cc:729] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.562173 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c6dcd194e148e3b8446a571a73eb69" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.562455 29661 raft_consensus.cc:3037] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.561903 29725 consensus_queue.cc:260] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.563548 29731 raft_consensus.cc:482] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.563766 29731 raft_consensus.cc:504] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.564517 29731 leader_election.cc:288] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 44483838ca544377b883694ab69979fe (127.28.136.65:45509)
I20240501 20:08:45.565337 29661 raft_consensus.cc:2454] T 87c6dcd194e148e3b8446a571a73eb69 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:45.565472 29725 ts_tablet_manager.cc:1398] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:08:45.566038 29468 leader_election.cc:302] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.566442 29468 leader_election.cc:302] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.566787 29732 raft_consensus.cc:2781] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.567868 29768 raft_consensus.cc:2781] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.567921 29725 tablet_bootstrap.cc:492] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.569628 29725 tablet_bootstrap.cc:654] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.571620 29728 tablet_bootstrap.cc:492] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.571825 29728 ts_tablet_manager.cc:1367] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.000s
I20240501 20:08:45.571864 29401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6fffc18bfc40bcb0c2dbb4a87d5851" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.572019 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c6dcd194e148e3b8446a571a73eb69" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:45.572365 29398 raft_consensus.cc:3037] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.572608 29728 raft_consensus.cc:348] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.572818 29728 raft_consensus.cc:374] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.572898 29728 raft_consensus.cc:729] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.573088 29399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6fffc18bfc40bcb0c2dbb4a87d5851" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:45.573338 29768 raft_consensus.cc:686] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [term 1 LEADER]: Becoming Leader. State: Replica: 62f50c608e55425abb9d25821764451c, State: Running, Role: LEADER
I20240501 20:08:45.573700 29400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c6dcd194e148e3b8446a571a73eb69" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.573911 29768 consensus_queue.cc:237] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
W20240501 20:08:45.574784 29467 leader_election.cc:341] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44483838ca544377b883694ab69979fe (127.28.136.65:45509): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.575131 29732 raft_consensus.cc:686] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [term 1 LEADER]: Becoming Leader. State: Replica: 62f50c608e55425abb9d25821764451c, State: Running, Role: LEADER
W20240501 20:08:45.575364 29467 leader_election.cc:341] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44483838ca544377b883694ab69979fe (127.28.136.65:45509): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.575412 29732 consensus_queue.cc:237] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.575771 29398 raft_consensus.cc:2454] T 87c6dcd194e148e3b8446a571a73eb69 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:45.578619 29728 consensus_queue.cc:260] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.579280 29725 tablet_bootstrap.cc:492] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.579409 29725 ts_tablet_manager.cc:1367] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20240501 20:08:45.580132 29725 raft_consensus.cc:348] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.580325 29725 raft_consensus.cc:374] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.580420 29725 raft_consensus.cc:729] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.580653 29725 consensus_queue.cc:260] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.581000 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4d209d9a14ff594d4014b38d589b4" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.582351 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4d209d9a14ff594d4014b38d589b4" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
W20240501 20:08:45.583114 29337 leader_election.cc:341] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.583393 29728 ts_tablet_manager.cc:1398] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:08:45.583261 29531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844da4163c3a440cbc20bae219ea7edd" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.583546 29725 ts_tablet_manager.cc:1398] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:08:45.583614 29531 raft_consensus.cc:2454] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.583688 29728 tablet_bootstrap.cc:492] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.583781 29725 tablet_bootstrap.cc:492] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
W20240501 20:08:45.584043 29336 leader_election.cc:341] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62f50c608e55425abb9d25821764451c (127.28.136.66:38189): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.584177 29336 leader_election.cc:302] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44483838ca544377b883694ab69979fe; no voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c
I20240501 20:08:45.584923 29336 leader_election.cc:302] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.585275 29734 raft_consensus.cc:2726] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:45.585278 29728 tablet_bootstrap.cc:654] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.585309 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844da4163c3a440cbc20bae219ea7edd" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.585436 29734 raft_consensus.cc:2781] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.585521 29734 raft_consensus.cc:482] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.585608 29734 raft_consensus.cc:3037] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:08:45.585888 29337 leader_election.cc:341] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.588512 29734 raft_consensus.cc:504] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.591037 29728 tablet_bootstrap.cc:492] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.591037 29734 leader_election.cc:288] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Requested vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.591202 29728 ts_tablet_manager.cc:1367] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:08:45.591936 29531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844da4163c3a440cbc20bae219ea7edd" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:45.592017 29728 raft_consensus.cc:348] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.592146 29531 raft_consensus.cc:3037] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.592260 29728 raft_consensus.cc:374] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.592334 29728 raft_consensus.cc:729] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.592582 29728 consensus_queue.cc:260] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.593009 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844da4163c3a440cbc20bae219ea7edd" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.593184 29725 tablet_bootstrap.cc:654] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
W20240501 20:08:45.593497 29337 leader_election.cc:341] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.595422 29734 raft_consensus.cc:482] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.595597 29734 raft_consensus.cc:504] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.596303 29734 leader_election.cc:288] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.596725 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98260be534042778599a404180811a1" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.597108 29661 raft_consensus.cc:2454] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.597576 29337 leader_election.cc:302] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe; no voters:
I20240501 20:08:45.598019 29734 raft_consensus.cc:2781] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.598161 29734 raft_consensus.cc:482] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.598255 29734 raft_consensus.cc:3037] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.599308 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98260be534042778599a404180811a1" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.599788 29529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195b8d506a36424b8c734e9b998ff7cb" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.600160 29529 raft_consensus.cc:2454] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 0.
W20240501 20:08:45.600682 29336 leader_election.cc:341] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62f50c608e55425abb9d25821764451c (127.28.136.66:38189): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.601193 29531 raft_consensus.cc:2454] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.600349 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6fffc18bfc40bcb0c2dbb4a87d5851" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.601534 29336 leader_election.cc:302] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters: 17b77e4a12564b67ab2849497540a431
I20240501 20:08:45.601910 29778 raft_consensus.cc:2781] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.602974 29728 ts_tablet_manager.cc:1398] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240501 20:08:45.602352 29258 catalog_manager.cc:5484] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f50c608e55425abb9d25821764451c (127.28.136.66). New cstate: current_term: 1 leader_uuid: "62f50c608e55425abb9d25821764451c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.603292 29728 tablet_bootstrap.cc:492] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.603852 29258 catalog_manager.cc:5484] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f50c608e55425abb9d25821764451c (127.28.136.66). New cstate: current_term: 1 leader_uuid: "62f50c608e55425abb9d25821764451c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.605146 29728 tablet_bootstrap.cc:654] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.605402 29731 raft_consensus.cc:482] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.605594 29731 raft_consensus.cc:504] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.606426 29731 leader_election.cc:288] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 62f50c608e55425abb9d25821764451c (127.28.136.66:38189)
I20240501 20:08:45.607673 29734 raft_consensus.cc:504] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.611239 29734 leader_election.cc:288] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Requested vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.612017 29734 raft_consensus.cc:482] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.612373 29734 raft_consensus.cc:504] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.614615 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0a8b308c8c46188e04cbc0d7605b34" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.614854 29530 raft_consensus.cc:2454] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 0.
I20240501 20:08:45.615321 29725 tablet_bootstrap.cc:492] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.615298 29734 leader_election.cc:288] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.615720 29734 raft_consensus.cc:482] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.616055 29734 raft_consensus.cc:504] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.616914 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be4b9ce1050425eb8483d182ad0fb01" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.617134 29662 raft_consensus.cc:2454] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.617062 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98260be534042778599a404180811a1" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.617369 29661 raft_consensus.cc:3037] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.620344 29531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be4b9ce1050425eb8483d182ad0fb01" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.620560 29531 raft_consensus.cc:2454] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.620838 29661 raft_consensus.cc:2454] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.620966 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98260be534042778599a404180811a1" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:45.620524 29778 raft_consensus.cc:686] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [term 1 LEADER]: Becoming Leader. State: Replica: 44483838ca544377b883694ab69979fe, State: Running, Role: LEADER
I20240501 20:08:45.621830 29778 consensus_queue.cc:237] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.620921 29336 leader_election.cc:302] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.622215 29728 tablet_bootstrap.cc:492] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.622355 29728 ts_tablet_manager.cc:1367] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.006s
I20240501 20:08:45.622311 29597 leader_election.cc:302] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.622808 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8b4b93b0d341279ffabb59631d3b84" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.622910 29734 leader_election.cc:288] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.622992 29530 raft_consensus.cc:2454] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.623082 29734 raft_consensus.cc:2781] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.623152 29734 raft_consensus.cc:482] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.623109 29728 raft_consensus.cc:348] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.623239 29734 raft_consensus.cc:3037] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.623286 29728 raft_consensus.cc:374] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.623368 29728 raft_consensus.cc:729] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.623579 29728 consensus_queue.cc:260] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.623898 29731 raft_consensus.cc:2781] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.624018 29731 raft_consensus.cc:482] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.624105 29731 raft_consensus.cc:3037] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.624148 29728 ts_tablet_manager.cc:1398] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:08:45.624390 29728 tablet_bootstrap.cc:492] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.625944 29728 tablet_bootstrap.cc:654] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.627133 29731 raft_consensus.cc:504] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.627830 29731 leader_election.cc:288] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 election: Requested vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 44483838ca544377b883694ab69979fe (127.28.136.65:45509)
I20240501 20:08:45.628333 29597 leader_election.cc:302] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c; no voters:
W20240501 20:08:45.628522 29336 leader_election.cc:341] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 62f50c608e55425abb9d25821764451c (127.28.136.66:38189): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.628806 29336 leader_election.cc:302] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.628965 29731 raft_consensus.cc:2781] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.629070 29731 raft_consensus.cc:482] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.629108 29783 raft_consensus.cc:2781] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.629176 29731 raft_consensus.cc:3037] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.629200 29783 raft_consensus.cc:482] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.629297 29783 raft_consensus.cc:3037] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.631978 29731 raft_consensus.cc:504] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.632105 29783 raft_consensus.cc:504] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.632423 29781 raft_consensus.cc:482] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.632642 29731 leader_election.cc:288] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 62f50c608e55425abb9d25821764451c (127.28.136.66:38189)
I20240501 20:08:45.632576 29781 raft_consensus.cc:504] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:45.633203 29783 leader_election.cc:288] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Requested vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.633425 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8b4b93b0d341279ffabb59631d3b84" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:45.633625 29530 raft_consensus.cc:3037] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.635577 29734 raft_consensus.cc:504] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.636934 29337 leader_election.cc:302] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe; no voters: 62f50c608e55425abb9d25821764451c
I20240501 20:08:45.637152 29531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be4b9ce1050425eb8483d182ad0fb01" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:45.637267 29783 raft_consensus.cc:2781] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.637341 29531 raft_consensus.cc:3037] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.637398 29783 raft_consensus.cc:686] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [term 1 LEADER]: Becoming Leader. State: Replica: 44483838ca544377b883694ab69979fe, State: Running, Role: LEADER
I20240501 20:08:45.637666 29783 consensus_queue.cc:237] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.639223 29661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8b4b93b0d341279ffabb59631d3b84" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.639874 29531 raft_consensus.cc:2454] T 7be4b9ce1050425eb8483d182ad0fb01 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.640202 29529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195b8d506a36424b8c734e9b998ff7cb" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:45.640368 29529 raft_consensus.cc:3037] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:08:45.641664 29337 leader_election.cc:341] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.615515 29725 ts_tablet_manager.cc:1367] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.005s
I20240501 20:08:45.643096 29529 raft_consensus.cc:2454] T 195b8d506a36424b8c734e9b998ff7cb P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 1.
I20240501 20:08:45.643462 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8b4b93b0d341279ffabb59631d3b84" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.643925 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be4b9ce1050425eb8483d182ad0fb01" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.644177 29660 raft_consensus.cc:3037] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.645618 29734 leader_election.cc:288] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Requested vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.646077 29726 tablet_bootstrap.cc:492] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.646266 29726 ts_tablet_manager.cc:1367] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.147s user 0.068s sys 0.051s
I20240501 20:08:45.646632 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6fffc18bfc40bcb0c2dbb4a87d5851" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.646831 29400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195b8d506a36424b8c734e9b998ff7cb" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.647002 29400 raft_consensus.cc:2454] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 0.
I20240501 20:08:45.647342 29726 raft_consensus.cc:348] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.647400 29400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0a8b308c8c46188e04cbc0d7605b34" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.647552 29726 raft_consensus.cc:374] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.647586 29400 raft_consensus.cc:2454] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 0.
I20240501 20:08:45.647632 29726 raft_consensus.cc:729] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.647186 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0a8b308c8c46188e04cbc0d7605b34" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:45.647781 29532 raft_consensus.cc:3037] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.647938 29400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195b8d506a36424b8c734e9b998ff7cb" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:45.647902 29726 consensus_queue.cc:260] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.648092 29400 raft_consensus.cc:3037] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.648440 29726 ts_tablet_manager.cc:1398] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.649403 29726 tablet_bootstrap.cc:492] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.649685 29728 tablet_bootstrap.cc:492] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.649834 29728 ts_tablet_manager.cc:1367] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.006s
I20240501 20:08:45.650369 29532 raft_consensus.cc:2454] T 0d0a8b308c8c46188e04cbc0d7605b34 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 1.
I20240501 20:08:45.650516 29728 raft_consensus.cc:348] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.650693 29728 raft_consensus.cc:374] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.650712 29400 raft_consensus.cc:2454] T 195b8d506a36424b8c734e9b998ff7cb P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 1.
I20240501 20:08:45.650763 29728 raft_consensus.cc:729] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.651054 29726 tablet_bootstrap.cc:654] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.651230 29400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0a8b308c8c46188e04cbc0d7605b34" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:45.651402 29400 raft_consensus.cc:3037] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.652563 29728 consensus_queue.cc:260] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.653360 29336 leader_election.cc:302] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.653705 29784 raft_consensus.cc:2781] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.653889 29784 raft_consensus.cc:686] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [term 1 LEADER]: Becoming Leader. State: Replica: 44483838ca544377b883694ab69979fe, State: Running, Role: LEADER
I20240501 20:08:45.654137 29400 raft_consensus.cc:2454] T 0d0a8b308c8c46188e04cbc0d7605b34 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 1.
I20240501 20:08:45.654407 29660 raft_consensus.cc:2454] T 7be4b9ce1050425eb8483d182ad0fb01 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.654237 29784 consensus_queue.cc:237] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.643858 29725 raft_consensus.cc:348] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.655051 29725 raft_consensus.cc:374] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.655088 29776 raft_consensus.cc:482] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.655218 29725 raft_consensus.cc:729] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.655222 29776 raft_consensus.cc:504] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.655577 29725 consensus_queue.cc:260] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.655881 29728 ts_tablet_manager.cc:1398] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:08:45.655903 29398 raft_consensus.cc:2454] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 0.
I20240501 20:08:45.656050 29776 leader_election.cc:288] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.656183 29728 tablet_bootstrap.cc:492] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c: Bootstrap starting.
I20240501 20:08:45.656476 29725 ts_tablet_manager.cc:1398] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 20:08:45.656878 29725 tablet_bootstrap.cc:492] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.657789 29728 tablet_bootstrap.cc:654] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.658684 29725 tablet_bootstrap.cc:654] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.660409 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dac71d185243e481b67806a29f7717" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.660902 29726 tablet_bootstrap.cc:492] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.661063 29726 ts_tablet_manager.cc:1367] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
W20240501 20:08:45.661161 29467 leader_election.cc:341] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44483838ca544377b883694ab69979fe (127.28.136.65:45509): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.661829 29726 raft_consensus.cc:348] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.636665 29530 raft_consensus.cc:2454] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.662176 29726 raft_consensus.cc:374] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.662282 29726 raft_consensus.cc:729] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.662354 29336 leader_election.cc:302] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.662421 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dac71d185243e481b67806a29f7717" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.662611 29734 raft_consensus.cc:2781] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.662542 29726 consensus_queue.cc:260] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
W20240501 20:08:45.662926 29468 leader_election.cc:341] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.663053 29468 leader_election.cc:302] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62f50c608e55425abb9d25821764451c; no voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe
I20240501 20:08:45.663439 29768 raft_consensus.cc:2726] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:45.663697 29726 ts_tablet_manager.cc:1398] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:45.663990 29726 tablet_bootstrap.cc:492] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.665478 29781 leader_election.cc:288] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141), 62f50c608e55425abb9d25821764451c (127.28.136.66:38189)
I20240501 20:08:45.666201 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0765346a16a14413bbeab82888bbc194" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.666363 29726 tablet_bootstrap.cc:654] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
W20240501 20:08:45.666743 29336 leader_election.cc:341] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 62f50c608e55425abb9d25821764451c (127.28.136.66:38189): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:08:45.668700 29337 leader_election.cc:341] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.668954 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0765346a16a14413bbeab82888bbc194" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.669088 29728 tablet_bootstrap.cc:492] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c: No bootstrap required, opened a new log
I20240501 20:08:45.669212 29728 ts_tablet_manager.cc:1367] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:08:45.669870 29597 leader_election.cc:302] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe; no voters:
I20240501 20:08:45.669972 29728 raft_consensus.cc:348] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:45.670153 29731 raft_consensus.cc:2781] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.670197 29728 raft_consensus.cc:374] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.670287 29728 raft_consensus.cc:729] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f50c608e55425abb9d25821764451c, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.670400 29258 catalog_manager.cc:5484] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 44483838ca544377b883694ab69979fe (127.28.136.65). New cstate: current_term: 1 leader_uuid: "44483838ca544377b883694ab69979fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.670640 29728 consensus_queue.cc:260] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:45.671723 29728 ts_tablet_manager.cc:1398] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:08:45.675577 29783 raft_consensus.cc:482] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.670172 29597 leader_election.cc:302] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe; no voters:
I20240501 20:08:45.675731 29783 raft_consensus.cc:504] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.676442 29783 leader_election.cc:288] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.676645 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06cf612821e4a1492c72f501e254a1a" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.676836 29530 raft_consensus.cc:2454] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.677142 29336 leader_election.cc:302] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.677361 29783 raft_consensus.cc:2781] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.677462 29783 raft_consensus.cc:482] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.677547 29783 raft_consensus.cc:3037] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.678033 29734 raft_consensus.cc:686] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [term 1 LEADER]: Becoming Leader. State: Replica: 44483838ca544377b883694ab69979fe, State: Running, Role: LEADER
I20240501 20:08:45.678382 29734 consensus_queue.cc:237] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.679807 29783 raft_consensus.cc:504] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.680451 29783 leader_election.cc:288] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Requested vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.680984 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06cf612821e4a1492c72f501e254a1a" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:45.681151 29530 raft_consensus.cc:3037] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.681663 29725 tablet_bootstrap.cc:492] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.681797 29725 ts_tablet_manager.cc:1367] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.003s
I20240501 20:08:45.681843 29258 catalog_manager.cc:5484] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 44483838ca544377b883694ab69979fe (127.28.136.65). New cstate: current_term: 1 leader_uuid: "44483838ca544377b883694ab69979fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.682375 29258 catalog_manager.cc:5484] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 44483838ca544377b883694ab69979fe (127.28.136.65). New cstate: current_term: 1 leader_uuid: "44483838ca544377b883694ab69979fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.682510 29725 raft_consensus.cc:348] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.682693 29725 raft_consensus.cc:374] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.682766 29725 raft_consensus.cc:729] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.682840 29258 catalog_manager.cc:5484] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 44483838ca544377b883694ab69979fe (127.28.136.65). New cstate: current_term: 1 leader_uuid: "44483838ca544377b883694ab69979fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.683004 29725 consensus_queue.cc:260] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.683538 29725 ts_tablet_manager.cc:1398] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.683722 29530 raft_consensus.cc:2454] T c06cf612821e4a1492c72f501e254a1a P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.683794 29725 tablet_bootstrap.cc:492] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.684320 29336 leader_election.cc:302] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.684569 29783 raft_consensus.cc:2781] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.684679 29783 raft_consensus.cc:686] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [term 1 LEADER]: Becoming Leader. State: Replica: 44483838ca544377b883694ab69979fe, State: Running, Role: LEADER
I20240501 20:08:45.684984 29783 consensus_queue.cc:237] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.685873 29768 raft_consensus.cc:482] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.686013 29768 raft_consensus.cc:504] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.686687 29725 tablet_bootstrap.cc:654] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.686718 29768 leader_election.cc:288] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.687085 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a92ab6a8ce24735a223363676c1bea6" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.687332 29398 raft_consensus.cc:2454] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:45.687880 29467 leader_election.cc:302] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.688143 29768 raft_consensus.cc:2781] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.688236 29768 raft_consensus.cc:482] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.688308 29768 raft_consensus.cc:3037] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.690110 29597 leader_election.cc:302] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe; no voters: 62f50c608e55425abb9d25821764451c
I20240501 20:08:45.690898 29800 raft_consensus.cc:2781] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.691097 29768 raft_consensus.cc:504] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.691540 29796 raft_consensus.cc:2741] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 20:08:45.692191 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a92ab6a8ce24735a223363676c1bea6" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:45.692377 29398 raft_consensus.cc:3037] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:08:45.692934 29337 leader_election.cc:341] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.693081 29337 leader_election.cc:302] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44483838ca544377b883694ab69979fe; no voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c
I20240501 20:08:45.693332 29734 raft_consensus.cc:2726] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:45.693722 29731 raft_consensus.cc:686] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [term 1 LEADER]: Becoming Leader. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Running, Role: LEADER
I20240501 20:08:45.694586 29800 raft_consensus.cc:686] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [term 1 LEADER]: Becoming Leader. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Running, Role: LEADER
I20240501 20:08:45.694240 29731 consensus_queue.cc:237] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.694994 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06cf612821e4a1492c72f501e254a1a" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.695238 29660 raft_consensus.cc:2454] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.695322 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06cf612821e4a1492c72f501e254a1a" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.695518 29662 raft_consensus.cc:3037] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.696166 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a92ab6a8ce24735a223363676c1bea6" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
W20240501 20:08:45.696774 29468 leader_election.cc:341] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.698240 29662 raft_consensus.cc:2454] T c06cf612821e4a1492c72f501e254a1a P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.698545 29768 leader_election.cc:288] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.698663 29726 tablet_bootstrap.cc:492] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.698853 29726 ts_tablet_manager.cc:1367] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.008s
I20240501 20:08:45.695510 29800 consensus_queue.cc:237] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.699631 29398 raft_consensus.cc:2454] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:45.699111 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a92ab6a8ce24735a223363676c1bea6" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.699672 29726 raft_consensus.cc:348] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.699892 29726 raft_consensus.cc:374] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.699987 29726 raft_consensus.cc:729] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.700194 29726 consensus_queue.cc:260] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
W20240501 20:08:45.700281 29468 leader_election.cc:341] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:08:45.700809 29726 ts_tablet_manager.cc:1398] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.700932 29467 leader_election.cc:302] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters: 17b77e4a12564b67ab2849497540a431
I20240501 20:08:45.701117 29726 tablet_bootstrap.cc:492] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.701398 29768 raft_consensus.cc:2781] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.701665 29768 raft_consensus.cc:686] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [term 1 LEADER]: Becoming Leader. State: Replica: 62f50c608e55425abb9d25821764451c, State: Running, Role: LEADER
I20240501 20:08:45.702039 29768 consensus_queue.cc:237] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.702476 29725 tablet_bootstrap.cc:492] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.702634 29725 ts_tablet_manager.cc:1367] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.001s
I20240501 20:08:45.702734 29726 tablet_bootstrap.cc:654] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.704360 29725 raft_consensus.cc:348] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.705281 29725 raft_consensus.cc:374] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.705559 29725 raft_consensus.cc:729] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.705978 29725 consensus_queue.cc:260] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.706594 29725 ts_tablet_manager.cc:1398] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:08:45.706876 29725 tablet_bootstrap.cc:492] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe: Bootstrap starting.
I20240501 20:08:45.708434 29725 tablet_bootstrap.cc:654] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.709534 29731 raft_consensus.cc:482] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.709570 29776 raft_consensus.cc:482] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.709661 29731 raft_consensus.cc:504] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.709728 29776 raft_consensus.cc:504] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.709856 29734 raft_consensus.cc:482] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.709990 29734 raft_consensus.cc:504] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.710260 29731 leader_election.cc:288] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 62f50c608e55425abb9d25821764451c (127.28.136.66:38189)
I20240501 20:08:45.710281 29776 leader_election.cc:288] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.710641 29734 leader_election.cc:288] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.710799 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844da4163c3a440cbc20bae219ea7edd" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.711621 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "844da4163c3a440cbc20bae219ea7edd" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.711687 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a41ea289d74d43a0218dafb4d4778c" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.711911 29662 raft_consensus.cc:2454] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:45.711938 29530 raft_consensus.cc:2379] T 844da4163c3a440cbc20bae219ea7edd P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17b77e4a12564b67ab2849497540a431 in current term 1: Already voted for candidate 44483838ca544377b883694ab69979fe in this term.
I20240501 20:08:45.712625 29468 leader_election.cc:302] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.712637 29597 leader_election.cc:302] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17b77e4a12564b67ab2849497540a431; no voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c
I20240501 20:08:45.713027 29776 raft_consensus.cc:2781] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.713053 29731 raft_consensus.cc:3037] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.713119 29776 raft_consensus.cc:482] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.713208 29776 raft_consensus.cc:3037] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.715786 29731 raft_consensus.cc:2726] T 844da4163c3a440cbc20bae219ea7edd P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:45.715983 29776 raft_consensus.cc:504] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.717154 29400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a41ea289d74d43a0218dafb4d4778c" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:45.717355 29400 raft_consensus.cc:3037] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.717695 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a41ea289d74d43a0218dafb4d4778c" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.717891 29660 raft_consensus.cc:3037] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.718030 29726 tablet_bootstrap.cc:492] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.718165 29726 ts_tablet_manager.cc:1367] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20240501 20:08:45.718945 29726 raft_consensus.cc:348] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.719146 29726 raft_consensus.cc:374] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.719237 29726 raft_consensus.cc:729] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.719491 29726 consensus_queue.cc:260] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.719990 29400 raft_consensus.cc:2454] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:45.720064 29726 ts_tablet_manager.cc:1398] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.720320 29726 tablet_bootstrap.cc:492] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.720688 29467 leader_election.cc:302] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.720978 29732 raft_consensus.cc:2781] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.721138 29732 raft_consensus.cc:686] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [term 1 LEADER]: Becoming Leader. State: Replica: 62f50c608e55425abb9d25821764451c, State: Running, Role: LEADER
I20240501 20:08:45.721406 29732 consensus_queue.cc:237] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.721720 29725 tablet_bootstrap.cc:492] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe: No bootstrap required, opened a new log
I20240501 20:08:45.721906 29725 ts_tablet_manager.cc:1367] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 20:08:45.721971 29726 tablet_bootstrap.cc:654] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.722780 29725 raft_consensus.cc:348] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.723083 29725 raft_consensus.cc:374] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.723219 29725 raft_consensus.cc:729] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44483838ca544377b883694ab69979fe, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.723563 29725 consensus_queue.cc:260] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.724264 29725 ts_tablet_manager.cc:1398] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.724364 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf0d60db6f845b7ac0e1bc99ddfa4c9" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.724622 29530 raft_consensus.cc:2454] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.725081 29336 leader_election.cc:302] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.725389 29734 raft_consensus.cc:2781] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.725481 29734 raft_consensus.cc:482] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.725600 29734 raft_consensus.cc:3037] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.725790 29726 tablet_bootstrap.cc:492] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.725632 29258 catalog_manager.cc:5484] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17b77e4a12564b67ab2849497540a431 (127.28.136.67). New cstate: current_term: 1 leader_uuid: "17b77e4a12564b67ab2849497540a431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:45.726061 29726 ts_tablet_manager.cc:1367] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:08:45.726902 29662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf0d60db6f845b7ac0e1bc99ddfa4c9" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.727105 29662 raft_consensus.cc:2454] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 0.
I20240501 20:08:45.727589 29726 raft_consensus.cc:348] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.727785 29726 raft_consensus.cc:374] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.727808 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a41ea289d74d43a0218dafb4d4778c" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.727860 29726 raft_consensus.cc:729] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.728001 29398 raft_consensus.cc:2362] T 59a41ea289d74d43a0218dafb4d4778c P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1. Re-sending same reply.
I20240501 20:08:45.728063 29726 consensus_queue.cc:260] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.728363 29734 raft_consensus.cc:504] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.728499 29660 raft_consensus.cc:2454] T 59a41ea289d74d43a0218dafb4d4778c P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:45.728634 29726 ts_tablet_manager.cc:1398] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.728957 29726 tablet_bootstrap.cc:492] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.729000 29776 leader_election.cc:288] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.730526 29726 tablet_bootstrap.cc:654] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.731240 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf0d60db6f845b7ac0e1bc99ddfa4c9" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:45.731459 29530 raft_consensus.cc:3037] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.734349 29530 raft_consensus.cc:2454] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.734956 29336 leader_election.cc:302] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.735258 29793 raft_consensus.cc:2781] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.735419 29793 raft_consensus.cc:686] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [term 1 LEADER]: Becoming Leader. State: Replica: 44483838ca544377b883694ab69979fe, State: Running, Role: LEADER
I20240501 20:08:45.735765 29793 consensus_queue.cc:237] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.737776 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddf0d60db6f845b7ac0e1bc99ddfa4c9" candidate_uuid: "44483838ca544377b883694ab69979fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.737998 29734 leader_election.cc:288] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [CANDIDATE]: Term 1 election: Requested vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.738184 29660 raft_consensus.cc:3037] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.738751 29726 tablet_bootstrap.cc:492] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.738888 29726 ts_tablet_manager.cc:1367] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20240501 20:08:45.739580 29726 raft_consensus.cc:348] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.739792 29726 raft_consensus.cc:374] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.739895 29726 raft_consensus.cc:729] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.740151 29726 consensus_queue.cc:260] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.740806 29726 ts_tablet_manager.cc:1398] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:08:45.741060 29726 tablet_bootstrap.cc:492] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.742595 29726 tablet_bootstrap.cc:654] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.743353 29258 catalog_manager.cc:5484] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17b77e4a12564b67ab2849497540a431 (127.28.136.67). New cstate: current_term: 1 leader_uuid: "17b77e4a12564b67ab2849497540a431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:45.744652 29726 tablet_bootstrap.cc:492] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.744813 29726 ts_tablet_manager.cc:1367] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:08:45.744930 29660 raft_consensus.cc:2454] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44483838ca544377b883694ab69979fe in term 1.
I20240501 20:08:45.745400 29726 raft_consensus.cc:348] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:45.743295 29257 catalog_manager.cc:5484] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f50c608e55425abb9d25821764451c (127.28.136.66). New cstate: current_term: 1 leader_uuid: "62f50c608e55425abb9d25821764451c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.745594 29726 raft_consensus.cc:374] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.743773 29260 catalog_manager.cc:5484] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 44483838ca544377b883694ab69979fe (127.28.136.65). New cstate: current_term: 1 leader_uuid: "44483838ca544377b883694ab69979fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.745663 29726 raft_consensus.cc:729] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.745926 29726 consensus_queue.cc:260] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:45.746532 29726 ts_tablet_manager.cc:1398] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:45.746821 29726 tablet_bootstrap.cc:492] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431: Bootstrap starting.
I20240501 20:08:45.747977 29726 tablet_bootstrap.cc:654] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.750140 29726 tablet_bootstrap.cc:492] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431: No bootstrap required, opened a new log
I20240501 20:08:45.750269 29726 ts_tablet_manager.cc:1367] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:08:45.750909 29726 raft_consensus.cc:348] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.751104 29726 raft_consensus.cc:374] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.751194 29726 raft_consensus.cc:729] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.751385 29726 consensus_queue.cc:260] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.751824 29726 ts_tablet_manager.cc:1398] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 20:08:45.753726 29260 catalog_manager.cc:5484] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f50c608e55425abb9d25821764451c (127.28.136.66). New cstate: current_term: 1 leader_uuid: "62f50c608e55425abb9d25821764451c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.756454 29257 catalog_manager.cc:5484] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 44483838ca544377b883694ab69979fe (127.28.136.65). New cstate: current_term: 1 leader_uuid: "44483838ca544377b883694ab69979fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.836179 29768 raft_consensus.cc:482] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.836423 29768 raft_consensus.cc:504] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.837100 29768 leader_election.cc:288] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.837628 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03911cb24ebc476a943bb7bd06878319" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.837651 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03911cb24ebc476a943bb7bd06878319" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.837836 29398 raft_consensus.cc:2454] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:45.837854 29660 raft_consensus.cc:2454] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:45.838255 29467 leader_election.cc:302] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.838515 29768 raft_consensus.cc:2781] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.838605 29768 raft_consensus.cc:482] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.838701 29768 raft_consensus.cc:3037] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.840502 29768 raft_consensus.cc:504] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.841085 29768 leader_election.cc:288] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.841392 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03911cb24ebc476a943bb7bd06878319" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:45.841562 29398 raft_consensus.cc:3037] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.841522 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03911cb24ebc476a943bb7bd06878319" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.841696 29660 raft_consensus.cc:3037] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.842772 29800 raft_consensus.cc:482] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.842952 29800 raft_consensus.cc:504] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.843645 29800 leader_election.cc:288] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62f50c608e55425abb9d25821764451c (127.28.136.66:38189), 44483838ca544377b883694ab69979fe (127.28.136.65:45509)
I20240501 20:08:45.843725 29398 raft_consensus.cc:2454] T 03911cb24ebc476a943bb7bd06878319 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:45.843979 29660 raft_consensus.cc:2454] T 03911cb24ebc476a943bb7bd06878319 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:45.844188 29467 leader_election.cc:302] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.844264 29530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8b4b93b0d341279ffabb59631d3b84" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:45.844489 29768 raft_consensus.cc:2781] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.844491 29530 raft_consensus.cc:2379] T 0e8b4b93b0d341279ffabb59631d3b84 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17b77e4a12564b67ab2849497540a431 in current term 1: Already voted for candidate 44483838ca544377b883694ab69979fe in this term.
I20240501 20:08:45.844674 29768 raft_consensus.cc:686] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [term 1 LEADER]: Becoming Leader. State: Replica: 62f50c608e55425abb9d25821764451c, State: Running, Role: LEADER
I20240501 20:08:45.844642 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8b4b93b0d341279ffabb59631d3b84" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.844939 29768 consensus_queue.cc:237] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.845280 29597 leader_election.cc:302] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17b77e4a12564b67ab2849497540a431; no voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c
I20240501 20:08:45.845530 29800 raft_consensus.cc:3037] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.847785 29800 raft_consensus.cc:2726] T 0e8b4b93b0d341279ffabb59631d3b84 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:45.848348 29257 catalog_manager.cc:5484] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f50c608e55425abb9d25821764451c (127.28.136.66). New cstate: current_term: 1 leader_uuid: "62f50c608e55425abb9d25821764451c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:45.994521 29768 raft_consensus.cc:482] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:45.994700 29768 raft_consensus.cc:504] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.995359 29768 leader_election.cc:288] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.995909 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4d209d9a14ff594d4014b38d589b4" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:45.995889 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4d209d9a14ff594d4014b38d589b4" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:45.996130 29398 raft_consensus.cc:2454] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:45.996152 29660 raft_consensus.cc:2454] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:45.996536 29467 leader_election.cc:302] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:45.996812 29768 raft_consensus.cc:2781] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:45.996912 29768 raft_consensus.cc:482] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:45.996997 29768 raft_consensus.cc:3037] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.998865 29768 raft_consensus.cc:504] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:45.999364 29768 leader_election.cc:288] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:45.999748 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4d209d9a14ff594d4014b38d589b4" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:45.999778 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4d209d9a14ff594d4014b38d589b4" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:45.999899 29398 raft_consensus.cc:3037] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.999943 29660 raft_consensus.cc:3037] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.001643 29398 raft_consensus.cc:2454] T 1ba4d209d9a14ff594d4014b38d589b4 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:46.001705 29660 raft_consensus.cc:2454] T 1ba4d209d9a14ff594d4014b38d589b4 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:46.001993 29467 leader_election.cc:302] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:46.002261 29768 raft_consensus.cc:2781] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:46.002398 29768 raft_consensus.cc:686] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [term 1 LEADER]: Becoming Leader. State: Replica: 62f50c608e55425abb9d25821764451c, State: Running, Role: LEADER
I20240501 20:08:46.002671 29768 consensus_queue.cc:237] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:46.006243 29257 catalog_manager.cc:5484] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f50c608e55425abb9d25821764451c (127.28.136.66). New cstate: current_term: 1 leader_uuid: "62f50c608e55425abb9d25821764451c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:46.011034 29768 raft_consensus.cc:482] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:46.011188 29768 raft_consensus.cc:504] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:46.011672 29768 leader_election.cc:288] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:46.012030 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0765346a16a14413bbeab82888bbc194" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:46.012082 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0765346a16a14413bbeab82888bbc194" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:46.012279 29660 raft_consensus.cc:2454] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:46.012281 29398 raft_consensus.cc:2454] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 0.
I20240501 20:08:46.012676 29467 leader_election.cc:302] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:46.012964 29768 raft_consensus.cc:2781] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:46.013043 29768 raft_consensus.cc:482] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:46.013137 29768 raft_consensus.cc:3037] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.014875 29768 raft_consensus.cc:504] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:46.015365 29768 leader_election.cc:288] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:46.015722 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0765346a16a14413bbeab82888bbc194" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:46.015758 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0765346a16a14413bbeab82888bbc194" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431"
I20240501 20:08:46.015885 29398 raft_consensus.cc:3037] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.015913 29660 raft_consensus.cc:3037] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.017709 29660 raft_consensus.cc:2454] T 0765346a16a14413bbeab82888bbc194 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:46.017792 29398 raft_consensus.cc:2454] T 0765346a16a14413bbeab82888bbc194 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62f50c608e55425abb9d25821764451c in term 1.
I20240501 20:08:46.018090 29468 leader_election.cc:302] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 62f50c608e55425abb9d25821764451c; no voters:
I20240501 20:08:46.018349 29768 raft_consensus.cc:2781] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:46.018481 29768 raft_consensus.cc:686] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [term 1 LEADER]: Becoming Leader. State: Replica: 62f50c608e55425abb9d25821764451c, State: Running, Role: LEADER
I20240501 20:08:46.018729 29768 consensus_queue.cc:237] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } }
I20240501 20:08:46.021232 29257 catalog_manager.cc:5484] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c reported cstate change: term changed from 0 to 1, leader changed from <none> to 62f50c608e55425abb9d25821764451c (127.28.136.66). New cstate: current_term: 1 leader_uuid: "62f50c608e55425abb9d25821764451c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:46.046784 29768 raft_consensus.cc:482] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:46.046978 29768 raft_consensus.cc:504] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:46.047636 29768 leader_election.cc:288] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 17b77e4a12564b67ab2849497540a431 (127.28.136.67:45141)
I20240501 20:08:46.048161 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98260be534042778599a404180811a1" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b77e4a12564b67ab2849497540a431" is_pre_election: true
I20240501 20:08:46.048166 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c98260be534042778599a404180811a1" candidate_uuid: "62f50c608e55425abb9d25821764451c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:46.048445 29660 raft_consensus.cc:2379] T c98260be534042778599a404180811a1 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 62f50c608e55425abb9d25821764451c in current term 1: Already voted for candidate 44483838ca544377b883694ab69979fe in this term.
I20240501 20:08:46.048911 29468 leader_election.cc:302] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62f50c608e55425abb9d25821764451c; no voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe
I20240501 20:08:46.049194 29768 raft_consensus.cc:3037] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.051048 29768 raft_consensus.cc:2726] T c98260be534042778599a404180811a1 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:46.063380 29768 consensus_queue.cc:1035] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.066066 29793 consensus_queue.cc:1035] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.071424 29776 consensus_queue.cc:1035] T 87c6dcd194e148e3b8446a571a73eb69 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.073328 29781 consensus_queue.cc:1035] T c06cf612821e4a1492c72f501e254a1a P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.081897 29800 consensus_queue.cc:1035] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.088078 29800 consensus_queue.cc:1035] T 195b8d506a36424b8c734e9b998ff7cb P 17b77e4a12564b67ab2849497540a431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.101583 29398 raft_consensus.cc:3037] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.103996 29776 consensus_queue.cc:1035] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.108335 29773 consensus_queue.cc:1035] T bd6fffc18bfc40bcb0c2dbb4a87d5851 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.127671 29781 consensus_queue.cc:1035] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.131914 29734 consensus_queue.cc:1035] T 7be4b9ce1050425eb8483d182ad0fb01 P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.132623 29734 consensus_queue.cc:1035] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.139379 29781 consensus_queue.cc:1035] T ddf0d60db6f845b7ac0e1bc99ddfa4c9 P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.139843 29781 consensus_queue.cc:1035] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.141126 29781 consensus_queue.cc:1035] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.144188 29781 consensus_queue.cc:1035] T c98260be534042778599a404180811a1 P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.146096 29800 consensus_queue.cc:1035] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.148883 29773 consensus_queue.cc:1035] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.149070 29800 raft_consensus.cc:482] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:46.149017 29734 consensus_queue.cc:1035] T 0e8b4b93b0d341279ffabb59631d3b84 P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.149240 29800 raft_consensus.cc:504] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:46.149741 29800 leader_election.cc:288] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 62f50c608e55425abb9d25821764451c (127.28.136.66:38189)
I20240501 20:08:46.151098 29529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dac71d185243e481b67806a29f7717" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:46.151252 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dac71d185243e481b67806a29f7717" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:46.151319 29529 raft_consensus.cc:2454] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 0.
I20240501 20:08:46.151429 29398 raft_consensus.cc:2454] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 0.
I20240501 20:08:46.151928 29597 leader_election.cc:302] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe; no voters:
I20240501 20:08:46.152200 29731 raft_consensus.cc:2781] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:46.152335 29731 raft_consensus.cc:482] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:46.152439 29731 raft_consensus.cc:3037] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.154904 29731 raft_consensus.cc:504] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:46.155489 29731 leader_election.cc:288] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 election: Requested vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 62f50c608e55425abb9d25821764451c (127.28.136.66:38189)
I20240501 20:08:46.156658 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dac71d185243e481b67806a29f7717" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe"
I20240501 20:08:46.156893 29398 raft_consensus.cc:3037] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.156980 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dac71d185243e481b67806a29f7717" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c"
I20240501 20:08:46.157193 29532 raft_consensus.cc:3037] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.157910 29796 consensus_queue.cc:1035] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.159493 29731 raft_consensus.cc:482] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:46.159637 29398 raft_consensus.cc:2454] T 40dac71d185243e481b67806a29f7717 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 1.
I20240501 20:08:46.159618 29731 raft_consensus.cc:504] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:46.160143 29731 leader_election.cc:288] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44483838ca544377b883694ab69979fe (127.28.136.65:45509), 62f50c608e55425abb9d25821764451c (127.28.136.66:38189)
I20240501 20:08:46.161485 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a92ab6a8ce24735a223363676c1bea6" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44483838ca544377b883694ab69979fe" is_pre_election: true
I20240501 20:08:46.161593 29531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a92ab6a8ce24735a223363676c1bea6" candidate_uuid: "17b77e4a12564b67ab2849497540a431" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62f50c608e55425abb9d25821764451c" is_pre_election: true
I20240501 20:08:46.161733 29398 raft_consensus.cc:2379] T 2a92ab6a8ce24735a223363676c1bea6 P 44483838ca544377b883694ab69979fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17b77e4a12564b67ab2849497540a431 in current term 1: Already voted for candidate 62f50c608e55425abb9d25821764451c in this term.
I20240501 20:08:46.162451 29597 leader_election.cc:302] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b77e4a12564b67ab2849497540a431, 44483838ca544377b883694ab69979fe; no voters:
I20240501 20:08:46.163077 29731 raft_consensus.cc:2781] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:46.163136 29597 leader_election.cc:302] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17b77e4a12564b67ab2849497540a431; no voters: 44483838ca544377b883694ab69979fe, 62f50c608e55425abb9d25821764451c
I20240501 20:08:46.163318 29731 raft_consensus.cc:686] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [term 1 LEADER]: Becoming Leader. State: Replica: 17b77e4a12564b67ab2849497540a431, State: Running, Role: LEADER
I20240501 20:08:46.163610 29731 consensus_queue.cc:237] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } }
I20240501 20:08:46.163815 29796 raft_consensus.cc:3037] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:46.166811 29796 raft_consensus.cc:2726] T 2a92ab6a8ce24735a223363676c1bea6 P 17b77e4a12564b67ab2849497540a431 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:46.168284 29258 catalog_manager.cc:5484] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17b77e4a12564b67ab2849497540a431 (127.28.136.67). New cstate: current_term: 1 leader_uuid: "17b77e4a12564b67ab2849497540a431" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:46.170769 29776 consensus_queue.cc:1035] T 59a41ea289d74d43a0218dafb4d4778c P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.173053 29776 consensus_queue.cc:1035] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.177667 29772 consensus_queue.cc:1035] T 2a92ab6a8ce24735a223363676c1bea6 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.186861 29532 raft_consensus.cc:2454] T 40dac71d185243e481b67806a29f7717 P 62f50c608e55425abb9d25821764451c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b77e4a12564b67ab2849497540a431 in term 1.
I20240501 20:08:46.204275 29781 consensus_queue.cc:1035] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.207461 29781 consensus_queue.cc:1035] T 844da4163c3a440cbc20bae219ea7edd P 44483838ca544377b883694ab69979fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.230075 29772 consensus_queue.cc:1035] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.247727 29732 consensus_queue.cc:1035] T 03911cb24ebc476a943bb7bd06878319 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.409229 29776 consensus_queue.cc:1035] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.418085 29768 consensus_queue.cc:1035] T 1ba4d209d9a14ff594d4014b38d589b4 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.485935 29768 consensus_queue.cc:1035] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b77e4a12564b67ab2849497540a431" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.492588 29732 consensus_queue.cc:1035] T 0765346a16a14413bbeab82888bbc194 P 62f50c608e55425abb9d25821764451c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:46.558362 29847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:46.558812 29847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:08:46.563197 29796 consensus_queue.cc:1035] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44483838ca544377b883694ab69979fe" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 45509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:46.568122 29800 consensus_queue.cc:1035] T 40dac71d185243e481b67806a29f7717 P 17b77e4a12564b67ab2849497540a431 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62f50c608e55425abb9d25821764451c" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:46.568542 29847 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 374.24 ms
time per row: 0.18712 ms
I20240501 20:08:47.565415 29379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:47.569433 29512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:47.571877 29628 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.725855 29628 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.726156 29379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:47.726680 29512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:47.903218 29628 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:47.903538 29379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:47.906738 29512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:48.538573 29628 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:48.548935 29512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:48.553061 29379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:48.708353 29628 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:08:48.708843 29512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:08:48.709111 29379 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:08:48.856623 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 29319
I20240501 20:08:48.862905 29441 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:48.999363 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 29319
W20240501 20:08:49.016997 29468 connection.cc:535] server connection from 127.28.136.65:46897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:08:49.017680 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 29450
I20240501 20:08:49.022982 29572 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:49.146358 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 29450
W20240501 20:08:49.148814 29597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:45509: connect: Connection refused (error 111)
W20240501 20:08:49.150167 29597 consensus_peers.cc:485] T 0d0a8b308c8c46188e04cbc0d7605b34 P 17b77e4a12564b67ab2849497540a431 -> Peer 44483838ca544377b883694ab69979fe (127.28.136.65:45509): Couldn't send request to peer 44483838ca544377b883694ab69979fe. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:45509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:08:49.164273 29598 connection.cc:535] server connection from 127.28.136.66:59497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:08:49.164321 29245 connection.cc:535] server connection from 127.28.136.66:58277 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:08:49.165009 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 29580
I20240501 20:08:49.170202 29702 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:49.288926 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 29580
I20240501 20:08:49.305588 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 29229
I20240501 20:08:49.306627 29290 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:49.434096 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 29229
2024-05-01T20:08:49Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (10012 ms)
[----------] 1 test from TableRangeRebalancingTest (10012 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240501 20:08:49.482116 29217 test_util.cc:268] Using random seed: 235245931
I20240501 20:08:49.483928 29217 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:49.483990 29217 ts_itest-base.cc:113] --------------
I20240501 20:08:49.484038 29217 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:08:49.484089 29217 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:49.484149 29217 ts_itest-base.cc:116] --------------
2024-05-01T20:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:49Z Disabled control of system clock
I20240501 20:08:49.504138 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:41141
--webserver_interface=127.28.136.126
--webserver_port=0
--builtin_ntp_servers=127.28.136.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.136.126:41141
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:08:49.813127 30087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:49.813503 30087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:49.813642 30087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:49.821080 30087 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:49.821223 30087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:49.821293 30087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:49.821374 30087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:49.821439 30087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:49.830615 30087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.136.126:41141
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:41141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.136.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:49.831945 30087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:49.833681 30087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:49.844820 30092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:49.845196 30093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:49.845453 30095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:49.846107 30087 server_base.cc:1034] running on GCE node
I20240501 20:08:49.846850 30087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:49.848239 30087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:49.849436 30087 hybrid_clock.cc:648] HybridClock initialized: now 1714594129849382 us; error 50 us; skew 500 ppm
I20240501 20:08:49.849771 30087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:49.851931 30087 webserver.cc:460] Webserver started at http://127.28.136.126:42313/ using document root <none> and password file <none>
I20240501 20:08:49.852640 30087 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:49.852779 30087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:49.853120 30087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:49.855814 30087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "291eed9df3b5431aa3aff27201a36d1a"
format_stamp: "Formatted at 2024-05-01 20:08:49 on dist-test-slave-sx0n"
I20240501 20:08:49.856557 30087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "291eed9df3b5431aa3aff27201a36d1a"
format_stamp: "Formatted at 2024-05-01 20:08:49 on dist-test-slave-sx0n"
I20240501 20:08:49.861428 30087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240501 20:08:49.864535 30101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:49.865794 30087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:08:49.865950 30087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
uuid: "291eed9df3b5431aa3aff27201a36d1a"
format_stamp: "Formatted at 2024-05-01 20:08:49 on dist-test-slave-sx0n"
I20240501 20:08:49.866153 30087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:49.879098 30087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:49.879873 30087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:49.880137 30087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:49.896353 30087 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.126:41141
I20240501 20:08:49.896359 30152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.126:41141 every 8 connection(s)
I20240501 20:08:49.897661 30087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:49.901278 30153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:49.905910 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 30087
I20240501 20:08:49.906081 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:49.911059 30153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a: Bootstrap starting.
I20240501 20:08:49.914698 30153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:49.915753 30153 log.cc:824] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:49.918323 30153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a: No bootstrap required, opened a new log
I20240501 20:08:49.924674 30153 raft_consensus.cc:348] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291eed9df3b5431aa3aff27201a36d1a" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 41141 } }
I20240501 20:08:49.925011 30153 raft_consensus.cc:374] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:49.925114 30153 raft_consensus.cc:729] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291eed9df3b5431aa3aff27201a36d1a, State: Initialized, Role: FOLLOWER
I20240501 20:08:49.925669 30153 consensus_queue.cc:260] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291eed9df3b5431aa3aff27201a36d1a" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 41141 } }
I20240501 20:08:49.925848 30153 raft_consensus.cc:388] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:49.925962 30153 raft_consensus.cc:482] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:49.926095 30153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:49.928062 30153 raft_consensus.cc:504] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291eed9df3b5431aa3aff27201a36d1a" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 41141 } }
I20240501 20:08:49.928535 30153 leader_election.cc:302] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 291eed9df3b5431aa3aff27201a36d1a; no voters:
I20240501 20:08:49.929188 30153 leader_election.cc:288] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:49.929323 30158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:49.929900 30158 raft_consensus.cc:686] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [term 1 LEADER]: Becoming Leader. State: Replica: 291eed9df3b5431aa3aff27201a36d1a, State: Running, Role: LEADER
I20240501 20:08:49.930402 30158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291eed9df3b5431aa3aff27201a36d1a" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 41141 } }
I20240501 20:08:49.931023 30153 sys_catalog.cc:564] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:49.933051 30159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "291eed9df3b5431aa3aff27201a36d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291eed9df3b5431aa3aff27201a36d1a" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 41141 } } }
I20240501 20:08:49.933353 30159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:49.933831 30160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 291eed9df3b5431aa3aff27201a36d1a. Latest consensus state: current_term: 1 leader_uuid: "291eed9df3b5431aa3aff27201a36d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291eed9df3b5431aa3aff27201a36d1a" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 41141 } } }
I20240501 20:08:49.934398 30160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:49.935752 30164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:49.940205 30164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:49.948228 30164 catalog_manager.cc:1347] Generated new cluster ID: 5e1edd41eccd405cbd97c23749efd779
I20240501 20:08:49.948347 30164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:49.962399 30164 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:49.964085 30164 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:49.987430 30164 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 291eed9df3b5431aa3aff27201a36d1a: Generated new TSK 0
I20240501 20:08:49.988513 30164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:49.998683 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.65:0
--local_ip_for_outbound_sockets=127.28.136.65
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--builtin_ntp_servers=127.28.136.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:08:50.309345 30177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:50.309628 30177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:50.309722 30177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:50.317068 30177 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:50.317195 30177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:50.317322 30177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.65
I20240501 20:08:50.326325 30177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.65
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:50.327605 30177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:50.329326 30177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:50.340960 30182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:50.341545 30183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:50.341854 30185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:50.342314 30177 server_base.cc:1034] running on GCE node
I20240501 20:08:50.342893 30177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:50.343762 30177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:50.344945 30177 hybrid_clock.cc:648] HybridClock initialized: now 1714594130344894 us; error 51 us; skew 500 ppm
I20240501 20:08:50.345228 30177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:50.347324 30177 webserver.cc:460] Webserver started at http://127.28.136.65:44137/ using document root <none> and password file <none>
I20240501 20:08:50.348006 30177 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:50.348114 30177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:50.348413 30177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:50.350883 30177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f323d17a50644b1cbadc716bf7419b68"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-sx0n"
I20240501 20:08:50.351613 30177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f323d17a50644b1cbadc716bf7419b68"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-sx0n"
I20240501 20:08:50.356434 30177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240501 20:08:50.359513 30191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:50.360719 30177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 20:08:50.360898 30177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f323d17a50644b1cbadc716bf7419b68"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-sx0n"
I20240501 20:08:50.361083 30177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:50.375587 30177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:50.376355 30177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:50.376626 30177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:50.377549 30177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:50.378973 30177 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:50.379065 30177 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:50.379168 30177 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:50.379235 30177 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:50.407754 30177 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.65:33613
I20240501 20:08:50.407811 30303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.65:33613 every 8 connection(s)
I20240501 20:08:50.409058 30177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:50.414070 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 30177
I20240501 20:08:50.414273 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:50.417383 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.66:0
--local_ip_for_outbound_sockets=127.28.136.66
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--builtin_ntp_servers=127.28.136.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:50.421773 30304 heartbeater.cc:347] Connected to a master server at 127.28.136.126:41141
I20240501 20:08:50.422138 30304 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:50.423197 30304 heartbeater.cc:511] Master 127.28.136.126:41141 requested a full tablet report, sending...
I20240501 20:08:50.425402 30118 ts_manager.cc:194] Registered new tserver with Master: f323d17a50644b1cbadc716bf7419b68 (127.28.136.65:33613)
I20240501 20:08:50.426924 30118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.65:42559
W20240501 20:08:50.727650 30307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:50.727943 30307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:50.728071 30307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:50.735385 30307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:50.735520 30307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:50.735672 30307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.66
I20240501 20:08:50.744560 30307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.66
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:50.745862 30307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:50.747538 30307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:50.758704 30312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:50.759151 30313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:50.759443 30315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:50.760135 30307 server_base.cc:1034] running on GCE node
I20240501 20:08:50.760768 30307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:50.761706 30307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:50.762898 30307 hybrid_clock.cc:648] HybridClock initialized: now 1714594130762850 us; error 38 us; skew 500 ppm
I20240501 20:08:50.763203 30307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:50.765408 30307 webserver.cc:460] Webserver started at http://127.28.136.66:37173/ using document root <none> and password file <none>
I20240501 20:08:50.766186 30307 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:50.766295 30307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:50.766623 30307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:50.769280 30307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d27929b976e640bab7d20975a7315ff0"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-sx0n"
I20240501 20:08:50.770102 30307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d27929b976e640bab7d20975a7315ff0"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-sx0n"
I20240501 20:08:50.775266 30307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:50.778625 30321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:50.779975 30307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:50.780141 30307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d27929b976e640bab7d20975a7315ff0"
format_stamp: "Formatted at 2024-05-01 20:08:50 on dist-test-slave-sx0n"
I20240501 20:08:50.780339 30307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:50.804580 30307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:50.805464 30307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:50.805728 30307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:50.806689 30307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:50.808176 30307 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:50.808270 30307 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:50.808377 30307 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:50.808449 30307 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:50.837404 30307 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.66:44741
I20240501 20:08:50.837453 30433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.66:44741 every 8 connection(s)
I20240501 20:08:50.838747 30307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:50.842835 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 30307
I20240501 20:08:50.842988 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:50.846238 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:0
--local_ip_for_outbound_sockets=127.28.136.67
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--builtin_ntp_servers=127.28.136.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:50.852028 30434 heartbeater.cc:347] Connected to a master server at 127.28.136.126:41141
I20240501 20:08:50.852387 30434 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:50.853446 30434 heartbeater.cc:511] Master 127.28.136.126:41141 requested a full tablet report, sending...
I20240501 20:08:50.855072 30118 ts_manager.cc:194] Registered new tserver with Master: d27929b976e640bab7d20975a7315ff0 (127.28.136.66:44741)
I20240501 20:08:50.855950 30118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.66:32855
W20240501 20:08:51.171231 30438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:51.171530 30438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:51.171645 30438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:51.179168 30438 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:51.179299 30438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:51.179450 30438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:08:51.188726 30438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:51.190063 30438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:51.191808 30438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:51.203642 30446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.204139 30444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.204453 30443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:51.204998 30438 server_base.cc:1034] running on GCE node
I20240501 20:08:51.205581 30438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:51.206573 30438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:51.207768 30438 hybrid_clock.cc:648] HybridClock initialized: now 1714594131207720 us; error 48 us; skew 500 ppm
I20240501 20:08:51.208072 30438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:51.210278 30438 webserver.cc:460] Webserver started at http://127.28.136.67:39843/ using document root <none> and password file <none>
I20240501 20:08:51.210988 30438 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:51.211108 30438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:51.211412 30438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:51.214084 30438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-sx0n"
I20240501 20:08:51.214818 30438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-sx0n"
I20240501 20:08:51.219765 30438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:08:51.223165 30452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.224551 30438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:51.224720 30438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-sx0n"
I20240501 20:08:51.224957 30438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:51.260062 30438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:51.261003 30438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:51.261281 30438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:51.262274 30438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:51.263785 30438 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:51.263881 30438 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.263988 30438 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:51.264056 30438 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.293414 30438 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:40307
I20240501 20:08:51.293483 30564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:40307 every 8 connection(s)
I20240501 20:08:51.294770 30438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:51.305176 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 30438
I20240501 20:08:51.305377 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:51.307030 30565 heartbeater.cc:347] Connected to a master server at 127.28.136.126:41141
I20240501 20:08:51.307374 30565 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:51.308264 30565 heartbeater.cc:511] Master 127.28.136.126:41141 requested a full tablet report, sending...
I20240501 20:08:51.308538 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.68:0
--local_ip_for_outbound_sockets=127.28.136.68
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--builtin_ntp_servers=127.28.136.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:51.309777 30118 ts_manager.cc:194] Registered new tserver with Master: 02f4a9a5969d411b8fa4fd2b9d2cbac4 (127.28.136.67:40307)
I20240501 20:08:51.310556 30118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:41185
I20240501 20:08:51.430298 30304 heartbeater.cc:503] Master 127.28.136.126:41141 was elected leader, sending a full tablet report...
W20240501 20:08:51.628176 30568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:51.628508 30568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:51.628652 30568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:51.636248 30568 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:51.636384 30568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:51.636543 30568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.68
I20240501 20:08:51.645677 30568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.68
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:51.647030 30568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:51.648767 30568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:51.661001 30573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.661293 30574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.661692 30576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:51.662335 30568 server_base.cc:1034] running on GCE node
I20240501 20:08:51.663188 30568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:51.664247 30568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:51.665488 30568 hybrid_clock.cc:648] HybridClock initialized: now 1714594131665438 us; error 41 us; skew 500 ppm
I20240501 20:08:51.665840 30568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:51.672425 30568 webserver.cc:460] Webserver started at http://127.28.136.68:43785/ using document root <none> and password file <none>
I20240501 20:08:51.673293 30568 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:51.673420 30568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:51.673751 30568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:51.676465 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5c5d8cfa5b954d3da253f373d1ae7235"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-sx0n"
I20240501 20:08:51.677300 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5c5d8cfa5b954d3da253f373d1ae7235"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-sx0n"
I20240501 20:08:51.682762 30568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:51.686445 30582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.687855 30568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:51.688019 30568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5c5d8cfa5b954d3da253f373d1ae7235"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-sx0n"
I20240501 20:08:51.688215 30568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:51.712843 30568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:51.713704 30568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:51.713965 30568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:51.714882 30568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:51.716323 30568 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:51.716416 30568 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.716524 30568 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:51.716601 30568 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.745255 30568 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.68:41945
I20240501 20:08:51.745307 30694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.68:41945 every 8 connection(s)
I20240501 20:08:51.746575 30568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:51.754504 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 30568
I20240501 20:08:51.754695 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:51.758074 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.69:0
--local_ip_for_outbound_sockets=127.28.136.69
--webserver_interface=127.28.136.69
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--builtin_ntp_servers=127.28.136.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:51.759495 30695 heartbeater.cc:347] Connected to a master server at 127.28.136.126:41141
I20240501 20:08:51.759799 30695 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:51.760613 30695 heartbeater.cc:511] Master 127.28.136.126:41141 requested a full tablet report, sending...
I20240501 20:08:51.762100 30118 ts_manager.cc:194] Registered new tserver with Master: 5c5d8cfa5b954d3da253f373d1ae7235 (127.28.136.68:41945)
I20240501 20:08:51.763046 30118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.68:35867
I20240501 20:08:51.858536 30434 heartbeater.cc:503] Master 127.28.136.126:41141 was elected leader, sending a full tablet report...
W20240501 20:08:52.074653 30698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:52.074951 30698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:52.075090 30698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:52.082597 30698 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:52.082747 30698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:52.082922 30698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.69
I20240501 20:08:52.092178 30698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.136.69
--webserver_port=0
--tserver_master_addrs=127.28.136.126:41141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.69
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:52.093591 30698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:52.095289 30698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:52.106969 30703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.107570 30704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.108042 30706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:52.108510 30698 server_base.cc:1034] running on GCE node
I20240501 20:08:52.109099 30698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:52.109993 30698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:52.111179 30698 hybrid_clock.cc:648] HybridClock initialized: now 1714594132111134 us; error 37 us; skew 500 ppm
I20240501 20:08:52.111474 30698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:52.113680 30698 webserver.cc:460] Webserver started at http://127.28.136.69:37839/ using document root <none> and password file <none>
I20240501 20:08:52.114364 30698 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:52.114461 30698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:52.114743 30698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:52.117233 30698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3b5fad109b49419c91ab6c83d3a0bce3"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-sx0n"
I20240501 20:08:52.117929 30698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3b5fad109b49419c91ab6c83d3a0bce3"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-sx0n"
I20240501 20:08:52.122831 30698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:52.126135 30712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.127466 30698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:08:52.127617 30698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3b5fad109b49419c91ab6c83d3a0bce3"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-sx0n"
I20240501 20:08:52.127795 30698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:52.149895 30698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:52.150761 30698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:52.151005 30698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:52.151948 30698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:52.153506 30698 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:52.153601 30698 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.153687 30698 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:52.153739 30698 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.184517 30698 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.69:36941
I20240501 20:08:52.184571 30824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.69:36941 every 8 connection(s)
I20240501 20:08:52.185935 30698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:52.193996 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 30698
I20240501 20:08:52.194175 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:52.198637 30825 heartbeater.cc:347] Connected to a master server at 127.28.136.126:41141
I20240501 20:08:52.198930 30825 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:52.199761 30825 heartbeater.cc:511] Master 127.28.136.126:41141 requested a full tablet report, sending...
I20240501 20:08:52.201086 30118 ts_manager.cc:194] Registered new tserver with Master: 3b5fad109b49419c91ab6c83d3a0bce3 (127.28.136.69:36941)
I20240501 20:08:52.201853 30118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.69:54391
I20240501 20:08:52.210078 29217 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:52.225246 30118 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41678:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:08:52.269047 30500 tablet_service.cc:1467] Processing CreateTablet for tablet 3be2cf237b0f4b468bd912da378bea98 (DEFAULT_TABLE table=TestTable [id=77fc61f2e20b4f05a8a6b48f5744c0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:52.269977 30369 tablet_service.cc:1467] Processing CreateTablet for tablet 3be2cf237b0f4b468bd912da378bea98 (DEFAULT_TABLE table=TestTable [id=77fc61f2e20b4f05a8a6b48f5744c0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:52.270844 30500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be2cf237b0f4b468bd912da378bea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:52.271370 30369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be2cf237b0f4b468bd912da378bea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:52.271893 30239 tablet_service.cc:1467] Processing CreateTablet for tablet 3be2cf237b0f4b468bd912da378bea98 (DEFAULT_TABLE table=TestTable [id=77fc61f2e20b4f05a8a6b48f5744c0dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:52.273288 30239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3be2cf237b0f4b468bd912da378bea98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:52.280812 30843 tablet_bootstrap.cc:492] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0: Bootstrap starting.
I20240501 20:08:52.283984 30843 tablet_bootstrap.cc:654] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:52.285074 30843 log.cc:824] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:52.289052 30845 tablet_bootstrap.cc:492] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68: Bootstrap starting.
I20240501 20:08:52.289606 30844 tablet_bootstrap.cc:492] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4: Bootstrap starting.
I20240501 20:08:52.292254 30845 tablet_bootstrap.cc:654] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:52.293411 30843 tablet_bootstrap.cc:492] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0: No bootstrap required, opened a new log
I20240501 20:08:52.293563 30845 log.cc:824] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:52.293684 30843 ts_tablet_manager.cc:1367] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240501 20:08:52.294245 30844 tablet_bootstrap.cc:654] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:52.295456 30844 log.cc:824] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:52.302192 30845 tablet_bootstrap.cc:492] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68: No bootstrap required, opened a new log
I20240501 20:08:52.302446 30845 ts_tablet_manager.cc:1367] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20240501 20:08:52.303426 30843 raft_consensus.cc:348] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.303822 30843 raft_consensus.cc:374] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:52.303920 30843 raft_consensus.cc:729] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d27929b976e640bab7d20975a7315ff0, State: Initialized, Role: FOLLOWER
I20240501 20:08:52.304708 30843 consensus_queue.cc:260] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.305891 30844 tablet_bootstrap.cc:492] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4: No bootstrap required, opened a new log
I20240501 20:08:52.306253 30844 ts_tablet_manager.cc:1367] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.002s
I20240501 20:08:52.310388 30845 raft_consensus.cc:348] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.310619 30843 ts_tablet_manager.cc:1398] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20240501 20:08:52.310719 30845 raft_consensus.cc:374] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:52.310827 30845 raft_consensus.cc:729] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f323d17a50644b1cbadc716bf7419b68, State: Initialized, Role: FOLLOWER
I20240501 20:08:52.311475 30845 consensus_queue.cc:260] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.313114 30845 ts_tablet_manager.cc:1398] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20240501 20:08:52.314023 30565 heartbeater.cc:503] Master 127.28.136.126:41141 was elected leader, sending a full tablet report...
I20240501 20:08:52.316049 30844 raft_consensus.cc:348] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.316480 30844 raft_consensus.cc:374] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:52.316591 30844 raft_consensus.cc:729] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f4a9a5969d411b8fa4fd2b9d2cbac4, State: Initialized, Role: FOLLOWER
I20240501 20:08:52.317409 30844 consensus_queue.cc:260] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.318852 30844 ts_tablet_manager.cc:1398] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
W20240501 20:08:52.341136 30435 tablet.cc:2369] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:52.367712 30851 raft_consensus.cc:482] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:52.367947 30851 raft_consensus.cc:504] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.369578 30851 leader_election.cc:288] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d27929b976e640bab7d20975a7315ff0 (127.28.136.66:44741), f323d17a50644b1cbadc716bf7419b68 (127.28.136.65:33613)
I20240501 20:08:52.376700 30259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be2cf237b0f4b468bd912da378bea98" candidate_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f323d17a50644b1cbadc716bf7419b68" is_pre_election: true
I20240501 20:08:52.376703 30389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be2cf237b0f4b468bd912da378bea98" candidate_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d27929b976e640bab7d20975a7315ff0" is_pre_election: true
I20240501 20:08:52.377183 30389 raft_consensus.cc:2454] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f4a9a5969d411b8fa4fd2b9d2cbac4 in term 0.
I20240501 20:08:52.377185 30259 raft_consensus.cc:2454] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f4a9a5969d411b8fa4fd2b9d2cbac4 in term 0.
I20240501 20:08:52.377732 30456 leader_election.cc:302] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f4a9a5969d411b8fa4fd2b9d2cbac4, f323d17a50644b1cbadc716bf7419b68; no voters:
I20240501 20:08:52.378084 30851 raft_consensus.cc:2781] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:52.378216 30851 raft_consensus.cc:482] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:52.378306 30851 raft_consensus.cc:3037] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:52.380226 30851 raft_consensus.cc:504] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.380846 30851 leader_election.cc:288] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [CANDIDATE]: Term 1 election: Requested vote from peers d27929b976e640bab7d20975a7315ff0 (127.28.136.66:44741), f323d17a50644b1cbadc716bf7419b68 (127.28.136.65:33613)
I20240501 20:08:52.381134 30389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be2cf237b0f4b468bd912da378bea98" candidate_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d27929b976e640bab7d20975a7315ff0"
I20240501 20:08:52.381304 30389 raft_consensus.cc:3037] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:52.381398 30259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be2cf237b0f4b468bd912da378bea98" candidate_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f323d17a50644b1cbadc716bf7419b68"
I20240501 20:08:52.381575 30259 raft_consensus.cc:3037] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:52.383347 30389 raft_consensus.cc:2454] T 3be2cf237b0f4b468bd912da378bea98 P d27929b976e640bab7d20975a7315ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f4a9a5969d411b8fa4fd2b9d2cbac4 in term 1.
I20240501 20:08:52.383613 30259 raft_consensus.cc:2454] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f4a9a5969d411b8fa4fd2b9d2cbac4 in term 1.
I20240501 20:08:52.383723 30455 leader_election.cc:302] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f4a9a5969d411b8fa4fd2b9d2cbac4, d27929b976e640bab7d20975a7315ff0; no voters:
I20240501 20:08:52.384006 30851 raft_consensus.cc:2781] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:52.384370 30851 raft_consensus.cc:686] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [term 1 LEADER]: Becoming Leader. State: Replica: 02f4a9a5969d411b8fa4fd2b9d2cbac4, State: Running, Role: LEADER
I20240501 20:08:52.384814 30851 consensus_queue.cc:237] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } }
I20240501 20:08:52.389446 30115 catalog_manager.cc:5484] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f4a9a5969d411b8fa4fd2b9d2cbac4 (127.28.136.67). New cstate: current_term: 1 leader_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f4a9a5969d411b8fa4fd2b9d2cbac4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 40307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 } health_report { overall_health: UNKNOWN } } }
W20240501 20:08:52.411926 30305 tablet.cc:2369] T 3be2cf237b0f4b468bd912da378bea98 P f323d17a50644b1cbadc716bf7419b68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:52.413375 29217 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:52.415926 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f323d17a50644b1cbadc716bf7419b68 to finish bootstrapping
I20240501 20:08:52.424348 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d27929b976e640bab7d20975a7315ff0 to finish bootstrapping
I20240501 20:08:52.431553 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 02f4a9a5969d411b8fa4fd2b9d2cbac4 to finish bootstrapping
I20240501 20:08:52.438905 29217 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5c5d8cfa5b954d3da253f373d1ae7235 to finish bootstrapping
I20240501 20:08:52.447357 29217 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3b5fad109b49419c91ab6c83d3a0bce3 to finish bootstrapping
W20240501 20:08:52.547217 30566 tablet.cc:2369] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:52.765581 30695 heartbeater.cc:503] Master 127.28.136.126:41141 was elected leader, sending a full tablet report...
I20240501 20:08:52.892673 30500 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:52.896404 30760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:52.897353 30369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:52.903736 30239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:52.909080 30630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:52.992616 30900 consensus_queue.cc:1035] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f323d17a50644b1cbadc716bf7419b68" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:53.001610 30900 consensus_queue.cc:1035] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d27929b976e640bab7d20975a7315ff0" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:53.073890 30760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.075670 30500 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:53.094421 30630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.094475 30369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.114555 30239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.204205 30825 heartbeater.cc:503] Master 127.28.136.126:41141 was elected leader, sending a full tablet report...
I20240501 20:08:53.285669 30500 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:53.289340 30760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.289744 30630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.303851 30369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.321035 30239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.905009 30500 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:53.907347 30760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.907716 30369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.920501 30239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:53.939193 30630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.088310 30500 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:54.090863 30760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.106628 30630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.110472 30369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.123157 30239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.266021 30760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.278489 30369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.279456 30239 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.281060 30630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:54.281705 30500 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:54.443082 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 30177
I20240501 20:08:54.444518 30299 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:54.538545 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 30177
W20240501 20:08:54.554131 30456 connection.cc:535] client connection to 127.28.136.65:33613 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:08:54.554412 30456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:08:54.554491 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 30307
I20240501 20:08:54.555377 30428 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:08:54.556190 30456 consensus_peers.cc:485] T 3be2cf237b0f4b468bd912da378bea98 P 02f4a9a5969d411b8fa4fd2b9d2cbac4 -> Peer f323d17a50644b1cbadc716bf7419b68 (127.28.136.65:33613): Couldn't send request to peer f323d17a50644b1cbadc716bf7419b68. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:54.650460 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 30307
I20240501 20:08:54.665782 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 30438
I20240501 20:08:54.666747 30560 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:54.769726 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 30438
I20240501 20:08:54.784924 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 30568
I20240501 20:08:54.785854 30689 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:54.875136 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 30568
I20240501 20:08:54.889462 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 30698
I20240501 20:08:54.890507 30820 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:54.985988 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 30698
I20240501 20:08:54.999749 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 30087
I20240501 20:08:55.000824 30148 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.114347 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 30087
2024-05-01T20:08:55Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (5670 ms)
[----------] 1 test from RebalanceStartSafetyTest (5670 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240501 20:08:55.152527 29217 test_util.cc:268] Using random seed: 240916329
I20240501 20:08:55.154346 29217 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:55.154434 29217 ts_itest-base.cc:113] --------------
I20240501 20:08:55.154482 29217 ts_itest-base.cc:114] 7 tablet servers
I20240501 20:08:55.154546 29217 ts_itest-base.cc:115] 5 replicas per TS
I20240501 20:08:55.154600 29217 ts_itest-base.cc:116] --------------
2024-05-01T20:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:55Z Disabled control of system clock
I20240501 20:08:55.175400 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:46583
--webserver_interface=127.28.136.126
--webserver_port=0
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.136.126:46583
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 20:08:55.484056 31086 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:08:55.484398 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:55.484517 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:55.484622 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:55.492064 31086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:55.492187 31086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:55.492280 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:55.492339 31086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:55.492424 31086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:55.501667 31086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.136.126:46583
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:46583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.136.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:55.503062 31086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:55.504886 31086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:55.515105 31091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:55.515515 31094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:55.516109 31092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:55.516645 31086 server_base.cc:1034] running on GCE node
I20240501 20:08:55.517401 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:55.518766 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:55.519966 31086 hybrid_clock.cc:648] HybridClock initialized: now 1714594135519905 us; error 35 us; skew 500 ppm
I20240501 20:08:55.520280 31086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:55.522505 31086 webserver.cc:460] Webserver started at http://127.28.136.126:42029/ using document root <none> and password file <none>
I20240501 20:08:55.523197 31086 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:55.523319 31086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:55.523612 31086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:55.526242 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "18e54ea4106c49c48e2a2a461e4657fd"
format_stamp: "Formatted at 2024-05-01 20:08:55 on dist-test-slave-sx0n"
I20240501 20:08:55.526955 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "18e54ea4106c49c48e2a2a461e4657fd"
format_stamp: "Formatted at 2024-05-01 20:08:55 on dist-test-slave-sx0n"
I20240501 20:08:55.531828 31086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240501 20:08:55.535072 31100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:55.536301 31086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:55.536458 31086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
uuid: "18e54ea4106c49c48e2a2a461e4657fd"
format_stamp: "Formatted at 2024-05-01 20:08:55 on dist-test-slave-sx0n"
I20240501 20:08:55.536635 31086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:55.555636 31086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:55.556480 31086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:55.556818 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:55.573232 31086 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.126:46583
I20240501 20:08:55.573254 31151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.126:46583 every 8 connection(s)
I20240501 20:08:55.574605 31086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:55.577190 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 31086
I20240501 20:08:55.577422 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:55.578886 31152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:55.590157 31152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd: Bootstrap starting.
I20240501 20:08:55.593235 31152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:55.594305 31152 log.cc:824] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:55.596516 31152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd: No bootstrap required, opened a new log
I20240501 20:08:55.602377 31152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e54ea4106c49c48e2a2a461e4657fd" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 46583 } }
I20240501 20:08:55.602677 31152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:55.602756 31152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e54ea4106c49c48e2a2a461e4657fd, State: Initialized, Role: FOLLOWER
I20240501 20:08:55.603282 31152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e54ea4106c49c48e2a2a461e4657fd" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 46583 } }
I20240501 20:08:55.603463 31152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:55.603551 31152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:55.603667 31152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:55.605800 31152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e54ea4106c49c48e2a2a461e4657fd" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 46583 } }
I20240501 20:08:55.606237 31152 leader_election.cc:302] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18e54ea4106c49c48e2a2a461e4657fd; no voters:
I20240501 20:08:55.606740 31152 leader_election.cc:288] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:55.606865 31157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:55.607399 31157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [term 1 LEADER]: Becoming Leader. State: Replica: 18e54ea4106c49c48e2a2a461e4657fd, State: Running, Role: LEADER
I20240501 20:08:55.607919 31157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e54ea4106c49c48e2a2a461e4657fd" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 46583 } }
I20240501 20:08:55.608508 31152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:55.610491 31159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e54ea4106c49c48e2a2a461e4657fd. Latest consensus state: current_term: 1 leader_uuid: "18e54ea4106c49c48e2a2a461e4657fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e54ea4106c49c48e2a2a461e4657fd" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 46583 } } }
I20240501 20:08:55.610469 31158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18e54ea4106c49c48e2a2a461e4657fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e54ea4106c49c48e2a2a461e4657fd" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 46583 } } }
I20240501 20:08:55.610785 31159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:55.610782 31158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:55.613644 31163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:55.618312 31163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:55.626226 31163 catalog_manager.cc:1347] Generated new cluster ID: 6afcd1fa33854b39902e6d8904d94ca6
I20240501 20:08:55.626322 31163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:55.661927 31163 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:55.663100 31163 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:55.677011 31163 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 18e54ea4106c49c48e2a2a461e4657fd: Generated new TSK 0
I20240501 20:08:55.677803 31163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:55.690711 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.65:0
--local_ip_for_outbound_sockets=127.28.136.65
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:08:56.006553 31176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:56.006872 31176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:56.006980 31176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:56.014518 31176 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:56.014652 31176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:56.014824 31176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.65
I20240501 20:08:56.023669 31176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.65
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:56.025024 31176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:56.026782 31176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:56.038201 31181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:56.038719 31184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:56.039125 31182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:56.039523 31176 server_base.cc:1034] running on GCE node
I20240501 20:08:56.040098 31176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:56.041040 31176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:56.042240 31176 hybrid_clock.cc:648] HybridClock initialized: now 1714594136042189 us; error 41 us; skew 500 ppm
I20240501 20:08:56.042528 31176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:56.044636 31176 webserver.cc:460] Webserver started at http://127.28.136.65:39527/ using document root <none> and password file <none>
I20240501 20:08:56.045341 31176 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:56.045447 31176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:56.045746 31176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:56.048252 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "341d3bb73a5c4e8f96176a281db4a60f"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.049026 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "341d3bb73a5c4e8f96176a281db4a60f"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.054016 31176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 20:08:56.057335 31190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.058633 31176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:56.058782 31176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "341d3bb73a5c4e8f96176a281db4a60f"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.058969 31176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:56.075050 31176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:56.075917 31176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:56.076179 31176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:56.077127 31176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:56.078558 31176 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:56.078662 31176 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.078769 31176 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:56.078837 31176 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.107600 31176 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.65:33401
I20240501 20:08:56.107646 31302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.65:33401 every 8 connection(s)
I20240501 20:08:56.108951 31176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:56.116524 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 31176
I20240501 20:08:56.116751 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:56.120355 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.66:0
--local_ip_for_outbound_sockets=127.28.136.66
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:56.121671 31303 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:08:56.122005 31303 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:56.123025 31303 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:08:56.125272 31117 ts_manager.cc:194] Registered new tserver with Master: 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401)
I20240501 20:08:56.126731 31117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.65:34881
W20240501 20:08:56.429852 31307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:56.430157 31307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:56.430290 31307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:56.437659 31307 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:56.437793 31307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:56.437944 31307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.66
I20240501 20:08:56.447270 31307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.66
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:56.448603 31307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:56.450330 31307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:56.462047 31312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:56.462327 31315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:56.462806 31313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:56.463407 31307 server_base.cc:1034] running on GCE node
I20240501 20:08:56.464051 31307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:56.465029 31307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:56.466240 31307 hybrid_clock.cc:648] HybridClock initialized: now 1714594136466191 us; error 38 us; skew 500 ppm
I20240501 20:08:56.466528 31307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:56.468575 31307 webserver.cc:460] Webserver started at http://127.28.136.66:46093/ using document root <none> and password file <none>
I20240501 20:08:56.469348 31307 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:56.469450 31307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:56.469725 31307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:56.472178 31307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "db07c3a323eb4a3c919eaf9b19b24986"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.472918 31307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "db07c3a323eb4a3c919eaf9b19b24986"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.477747 31307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 20:08:56.480966 31321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.482285 31307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:56.482422 31307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "db07c3a323eb4a3c919eaf9b19b24986"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.482599 31307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:56.511617 31307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:56.512462 31307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:56.512704 31307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:56.513659 31307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:56.515209 31307 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:56.515296 31307 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.515379 31307 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:56.515432 31307 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.544027 31307 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.66:42527
I20240501 20:08:56.544082 31433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.66:42527 every 8 connection(s)
I20240501 20:08:56.545387 31307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:56.545969 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 31307
I20240501 20:08:56.546203 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:56.549342 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:0
--local_ip_for_outbound_sockets=127.28.136.67
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:56.558825 31434 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:08:56.559134 31434 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:56.559934 31434 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:08:56.561381 31117 ts_manager.cc:194] Registered new tserver with Master: db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527)
I20240501 20:08:56.562144 31117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.66:55935
W20240501 20:08:56.868108 31437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:56.868405 31437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:56.868542 31437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:56.875847 31437 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:56.875986 31437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:56.876143 31437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:08:56.884945 31437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:56.886265 31437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:56.887959 31437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:56.898983 31442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:56.899533 31443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:56.900015 31445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:56.900592 31437 server_base.cc:1034] running on GCE node
I20240501 20:08:56.901158 31437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:56.902001 31437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:56.903203 31437 hybrid_clock.cc:648] HybridClock initialized: now 1714594136903161 us; error 39 us; skew 500 ppm
I20240501 20:08:56.903496 31437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:56.909889 31437 webserver.cc:460] Webserver started at http://127.28.136.67:44613/ using document root <none> and password file <none>
I20240501 20:08:56.910636 31437 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:56.910750 31437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:56.911058 31437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:56.913686 31437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.914412 31437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.919248 31437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:56.922500 31451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.923766 31437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:56.923913 31437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
format_stamp: "Formatted at 2024-05-01 20:08:56 on dist-test-slave-sx0n"
I20240501 20:08:56.924099 31437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:56.946019 31437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:56.946893 31437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:56.947168 31437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:56.948101 31437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:56.949599 31437 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:56.949694 31437 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.949798 31437 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:56.949863 31437 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:56.978438 31437 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:42361
I20240501 20:08:56.978482 31563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:42361 every 8 connection(s)
I20240501 20:08:56.979700 31437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:56.985643 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 31437
I20240501 20:08:56.985807 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:56.988886 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.68:0
--local_ip_for_outbound_sockets=127.28.136.68
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:56.991767 31564 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:08:56.992050 31564 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:56.992841 31564 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:08:56.994139 31117 ts_manager.cc:194] Registered new tserver with Master: ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:08:56.994891 31117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:53091
I20240501 20:08:57.129868 31303 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
W20240501 20:08:57.301162 31567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:57.301470 31567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:57.301570 31567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:57.309130 31567 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:57.309281 31567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:57.309432 31567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.68
I20240501 20:08:57.320820 31567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.68
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:57.322371 31567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:57.324146 31567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:57.336202 31572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:57.336627 31573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:57.337137 31575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:57.337671 31567 server_base.cc:1034] running on GCE node
I20240501 20:08:57.338227 31567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:57.339121 31567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:57.340312 31567 hybrid_clock.cc:648] HybridClock initialized: now 1714594137340266 us; error 37 us; skew 500 ppm
I20240501 20:08:57.340626 31567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:57.342809 31567 webserver.cc:460] Webserver started at http://127.28.136.68:35825/ using document root <none> and password file <none>
I20240501 20:08:57.343497 31567 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:57.343609 31567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:57.343904 31567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:57.346436 31567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-sx0n"
I20240501 20:08:57.347154 31567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-sx0n"
I20240501 20:08:57.351883 31567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240501 20:08:57.355078 31581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.356302 31567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 20:08:57.356444 31567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-sx0n"
I20240501 20:08:57.356627 31567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:57.371512 31567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:57.372282 31567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:57.372534 31567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:57.373427 31567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:57.374804 31567 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:57.374894 31567 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.375000 31567 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:57.375066 31567 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.403599 31567 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.68:44049
I20240501 20:08:57.403626 31693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.68:44049 every 8 connection(s)
I20240501 20:08:57.404958 31567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:57.413995 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 31567
I20240501 20:08:57.414188 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:57.417017 31694 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:08:57.417337 31694 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:57.417356 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.69:0
--local_ip_for_outbound_sockets=127.28.136.69
--webserver_interface=127.28.136.69
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:57.418321 31694 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:08:57.419699 31117 ts_manager.cc:194] Registered new tserver with Master: ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:08:57.420464 31117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.68:38311
I20240501 20:08:57.564664 31434 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
W20240501 20:08:57.724553 31697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:57.724865 31697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:57.724996 31697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:57.732304 31697 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:57.732415 31697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:57.732568 31697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.69
I20240501 20:08:57.741309 31697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.136.69
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.69
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:57.742637 31697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:57.744331 31697 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:57.755748 31702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:57.756145 31705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:57.756645 31703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:57.757218 31697 server_base.cc:1034] running on GCE node
I20240501 20:08:57.757742 31697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:57.758630 31697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:57.759835 31697 hybrid_clock.cc:648] HybridClock initialized: now 1714594137759775 us; error 55 us; skew 500 ppm
I20240501 20:08:57.760118 31697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:57.762173 31697 webserver.cc:460] Webserver started at http://127.28.136.69:44383/ using document root <none> and password file <none>
I20240501 20:08:57.762900 31697 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:57.763003 31697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:57.763288 31697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:57.765740 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d73d71e1531b4649b363236998019221"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-sx0n"
I20240501 20:08:57.766448 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d73d71e1531b4649b363236998019221"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-sx0n"
I20240501 20:08:57.771030 31697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:08:57.774106 31711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.775234 31697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 20:08:57.775372 31697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d73d71e1531b4649b363236998019221"
format_stamp: "Formatted at 2024-05-01 20:08:57 on dist-test-slave-sx0n"
I20240501 20:08:57.775540 31697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:57.792685 31697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:57.793448 31697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:57.793699 31697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:57.794520 31697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:57.795831 31697 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:57.795922 31697 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.796025 31697 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:57.796092 31697 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:57.824146 31697 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.69:43505
I20240501 20:08:57.824206 31824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.69:43505 every 8 connection(s)
I20240501 20:08:57.825409 31697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:57.833508 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 31697
I20240501 20:08:57.833664 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:57.836766 31825 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:08:57.836958 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.70:0
--local_ip_for_outbound_sockets=127.28.136.70
--webserver_interface=127.28.136.70
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:57.837036 31825 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:57.837785 31825 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:08:57.839033 31117 ts_manager.cc:194] Registered new tserver with Master: d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:08:57.839928 31117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.69:57109
I20240501 20:08:57.997118 31564 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
W20240501 20:08:58.144718 31828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:58.145053 31828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:58.145179 31828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:58.153512 31828 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:58.153633 31828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:58.153781 31828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.70
I20240501 20:08:58.162585 31828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.136.70
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.70
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:58.163903 31828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:58.165583 31828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:58.177147 31834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:58.177582 31836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:58.178157 31833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:58.178689 31828 server_base.cc:1034] running on GCE node
I20240501 20:08:58.179296 31828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:58.180258 31828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:58.181452 31828 hybrid_clock.cc:648] HybridClock initialized: now 1714594138181405 us; error 47 us; skew 500 ppm
I20240501 20:08:58.181751 31828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:58.183782 31828 webserver.cc:460] Webserver started at http://127.28.136.70:41877/ using document root <none> and password file <none>
I20240501 20:08:58.184489 31828 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:58.184593 31828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:58.184926 31828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:58.187580 31828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "582c7a51c0ac43b4934078ebdf434485"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-sx0n"
I20240501 20:08:58.188325 31828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "582c7a51c0ac43b4934078ebdf434485"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-sx0n"
I20240501 20:08:58.193153 31828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240501 20:08:58.196290 31842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.197568 31828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:58.197715 31828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "582c7a51c0ac43b4934078ebdf434485"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-sx0n"
I20240501 20:08:58.197893 31828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:58.219213 31828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:58.219995 31828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:58.220264 31828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:58.221185 31828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:58.222571 31828 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:58.222671 31828 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.222772 31828 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:58.222836 31828 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.251083 31828 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.70:37941
I20240501 20:08:58.251132 31954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.70:37941 every 8 connection(s)
I20240501 20:08:58.252380 31828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:08:58.262194 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 31828
I20240501 20:08:58.262683 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:08:58.264927 31955 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:08:58.265220 31955 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:58.265789 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.71:0
--local_ip_for_outbound_sockets=127.28.136.71
--webserver_interface=127.28.136.71
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:58.266003 31955 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:08:58.267315 31117 ts_manager.cc:194] Registered new tserver with Master: 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:08:58.268079 31117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.70:51697
I20240501 20:08:58.422951 31694 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
W20240501 20:08:58.584250 31958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:58.584569 31958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:58.584705 31958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:58.592195 31958 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:58.592332 31958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:58.592490 31958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.71
I20240501 20:08:58.601434 31958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.136.71
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.71
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:58.602753 31958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:58.604445 31958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:58.616040 31963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:58.616389 31964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:58.616984 31966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:58.617604 31958 server_base.cc:1034] running on GCE node
I20240501 20:08:58.618155 31958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:58.619033 31958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:58.620230 31958 hybrid_clock.cc:648] HybridClock initialized: now 1714594138620181 us; error 45 us; skew 500 ppm
I20240501 20:08:58.620532 31958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:58.622745 31958 webserver.cc:460] Webserver started at http://127.28.136.71:34655/ using document root <none> and password file <none>
I20240501 20:08:58.623476 31958 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:58.623589 31958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:58.623895 31958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:58.626518 31958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c42de85026a84ed7b4f462cf25433e01"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-sx0n"
I20240501 20:08:58.627285 31958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c42de85026a84ed7b4f462cf25433e01"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-sx0n"
I20240501 20:08:58.632373 31958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:08:58.635885 31972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.637403 31958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:58.637571 31958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c42de85026a84ed7b4f462cf25433e01"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-sx0n"
I20240501 20:08:58.637778 31958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:58.672180 31958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:58.673126 31958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:58.673418 31958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:58.674409 31958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:58.675930 31958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:58.676024 31958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.676144 31958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:58.676213 31958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:58.705165 31958 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.71:35749
I20240501 20:08:58.705220 32084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.71:35749 every 8 connection(s)
I20240501 20:08:58.706439 31958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:08:58.711273 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 31958
I20240501 20:08:58.711422 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:08:58.718561 32085 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:08:58.718848 32085 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:58.719628 32085 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:08:58.720957 31117 ts_manager.cc:194] Registered new tserver with Master: c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749)
I20240501 20:08:58.721709 31117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.71:52299
I20240501 20:08:58.727137 29217 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:08:58.742913 31117 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39828:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:08:58.793234 31629 tablet_service.cc:1467] Processing CreateTablet for tablet 750c72210b3240839bb156782aa1bdca (DEFAULT_TABLE table=TestTable [id=5168dc22b35340eb910bd1a9fe6598c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.794895 31629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750c72210b3240839bb156782aa1bdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.800318 31760 tablet_service.cc:1467] Processing CreateTablet for tablet 750c72210b3240839bb156782aa1bdca (DEFAULT_TABLE table=TestTable [id=5168dc22b35340eb910bd1a9fe6598c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.801908 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750c72210b3240839bb156782aa1bdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.802376 32020 tablet_service.cc:1467] Processing CreateTablet for tablet 750c72210b3240839bb156782aa1bdca (DEFAULT_TABLE table=TestTable [id=5168dc22b35340eb910bd1a9fe6598c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.803855 32020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750c72210b3240839bb156782aa1bdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.803555 31369 tablet_service.cc:1467] Processing CreateTablet for tablet 750c72210b3240839bb156782aa1bdca (DEFAULT_TABLE table=TestTable [id=5168dc22b35340eb910bd1a9fe6598c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.805013 31369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750c72210b3240839bb156782aa1bdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.812584 31238 tablet_service.cc:1467] Processing CreateTablet for tablet 750c72210b3240839bb156782aa1bdca (DEFAULT_TABLE table=TestTable [id=5168dc22b35340eb910bd1a9fe6598c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:58.814399 31238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750c72210b3240839bb156782aa1bdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:58.824352 32108 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
I20240501 20:08:58.828176 32108 tablet_bootstrap.cc:654] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.831665 32109 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:08:58.835779 32109 tablet_bootstrap.cc:654] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.838553 32106 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:08:58.841635 32107 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:08:58.842280 31825 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
I20240501 20:08:58.842717 32106 tablet_bootstrap.cc:654] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.844247 32106 log.cc:824] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:58.845278 32107 tablet_bootstrap.cc:654] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.846474 32107 log.cc:824] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:58.849260 32108 log.cc:824] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:58.850044 32109 log.cc:824] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:58.851079 32110 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:08:58.854873 32106 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:08:58.855206 32106 ts_tablet_manager.cc:1367] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20240501 20:08:58.855674 32107 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:08:58.855677 32109 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:08:58.855960 32109 ts_tablet_manager.cc:1367] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20240501 20:08:58.855962 32107 ts_tablet_manager.cc:1367] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20240501 20:08:58.856423 32110 tablet_bootstrap.cc:654] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:58.857839 32110 log.cc:824] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:58.862969 32108 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: No bootstrap required, opened a new log
I20240501 20:08:58.863335 32108 ts_tablet_manager.cc:1367] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.005s
I20240501 20:08:58.864214 32106 raft_consensus.cc:348] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.864712 32106 raft_consensus.cc:374] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.864853 32106 raft_consensus.cc:729] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.864711 32109 raft_consensus.cc:348] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.865161 32109 raft_consensus.cc:374] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.865275 32109 raft_consensus.cc:729] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.865737 32106 consensus_queue.cc:260] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.866039 32109 consensus_queue.cc:260] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.869374 32110 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:08:58.869762 32110 ts_tablet_manager.cc:1367] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.002s
I20240501 20:08:58.873082 32107 raft_consensus.cc:348] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.877113 32107 raft_consensus.cc:374] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.877236 32107 raft_consensus.cc:729] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.878105 32107 consensus_queue.cc:260] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.879007 32108 raft_consensus.cc:348] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.879483 32108 raft_consensus.cc:374] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.879585 32108 raft_consensus.cc:729] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.880316 32110 raft_consensus.cc:348] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.880467 32108 consensus_queue.cc:260] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.880861 32110 raft_consensus.cc:374] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:58.881004 32110 raft_consensus.cc:729] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:08:58.881795 32110 consensus_queue.cc:260] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.884615 32106 ts_tablet_manager.cc:1398] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.029s user 0.012s sys 0.017s
I20240501 20:08:58.885286 32109 ts_tablet_manager.cc:1398] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.029s user 0.010s sys 0.021s
I20240501 20:08:58.886647 32107 ts_tablet_manager.cc:1398] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.031s user 0.010s sys 0.004s
I20240501 20:08:58.886713 32110 ts_tablet_manager.cc:1398] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.017s user 0.007s sys 0.007s
I20240501 20:08:58.887607 32085 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
I20240501 20:08:58.890805 32108 ts_tablet_manager.cc:1398] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.027s user 0.009s sys 0.003s
W20240501 20:08:58.907416 31695 tablet.cc:2369] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:08:58.957976 32086 tablet.cc:2369] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:58.959389 32118 raft_consensus.cc:482] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:58.959640 32118 raft_consensus.cc:504] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.961630 32118 leader_election.cc:288] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:08:58.970496 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750c72210b3240839bb156782aa1bdca" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:08:58.971065 31649 raft_consensus.cc:2454] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:08:58.973109 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750c72210b3240839bb156782aa1bdca" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:08:58.973322 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750c72210b3240839bb156782aa1bdca" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:08:58.973699 31389 raft_consensus.cc:2454] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:08:58.973698 32040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750c72210b3240839bb156782aa1bdca" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42de85026a84ed7b4f462cf25433e01" is_pre_election: true
I20240501 20:08:58.973914 31258 raft_consensus.cc:2454] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:08:58.974197 32040 raft_consensus.cc:2454] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:08:58.974195 31715 leader_election.cc:302] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: ce9dba79a8c94a7b87bf51e9ed4b5fe1, d73d71e1531b4649b363236998019221, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:08:58.974627 32118 raft_consensus.cc:2781] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:58.974756 32118 raft_consensus.cc:482] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:58.974854 32118 raft_consensus.cc:3037] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.976904 32118 raft_consensus.cc:504] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.977653 32118 leader_election.cc:288] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 election: Requested vote from peers c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:08:58.977980 32040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750c72210b3240839bb156782aa1bdca" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42de85026a84ed7b4f462cf25433e01"
I20240501 20:08:58.978055 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750c72210b3240839bb156782aa1bdca" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:08:58.978181 32040 raft_consensus.cc:3037] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.978233 31389 raft_consensus.cc:3037] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.978371 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750c72210b3240839bb156782aa1bdca" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:08:58.978552 31258 raft_consensus.cc:3037] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.980998 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750c72210b3240839bb156782aa1bdca" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:08:58.981134 32040 raft_consensus.cc:2454] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 1.
I20240501 20:08:58.981202 31649 raft_consensus.cc:3037] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:58.981320 31389 raft_consensus.cc:2454] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 1.
I20240501 20:08:58.981812 31715 leader_election.cc:302] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: c42de85026a84ed7b4f462cf25433e01, d73d71e1531b4649b363236998019221, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:08:58.982116 32118 raft_consensus.cc:2781] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:58.982363 31258 raft_consensus.cc:2454] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 1.
I20240501 20:08:58.982573 32118 raft_consensus.cc:686] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 1 LEADER]: Becoming Leader. State: Replica: d73d71e1531b4649b363236998019221, State: Running, Role: LEADER
I20240501 20:08:58.983366 32118 consensus_queue.cc:237] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:08:58.983675 31649 raft_consensus.cc:2454] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 1.
I20240501 20:08:58.988507 31117 catalog_manager.cc:5484] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d71e1531b4649b363236998019221 (127.28.136.69). New cstate: current_term: 1 leader_uuid: "d73d71e1531b4649b363236998019221" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:59.006531 29217 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:08:59.008512 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 341d3bb73a5c4e8f96176a281db4a60f to finish bootstrapping
I20240501 20:08:59.016266 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver db07c3a323eb4a3c919eaf9b19b24986 to finish bootstrapping
I20240501 20:08:59.023304 29217 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ec65c12cd0724cf9aa16fcd61bbedae4 to finish bootstrapping
I20240501 20:08:59.031785 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ce9dba79a8c94a7b87bf51e9ed4b5fe1 to finish bootstrapping
I20240501 20:08:59.039146 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d73d71e1531b4649b363236998019221 to finish bootstrapping
I20240501 20:08:59.046305 29217 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 582c7a51c0ac43b4934078ebdf434485 to finish bootstrapping
W20240501 20:08:59.048650 31435 tablet.cc:2369] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:59.054641 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c42de85026a84ed7b4f462cf25433e01 to finish bootstrapping
I20240501 20:08:59.061702 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 31958
I20240501 20:08:59.062626 32080 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:08:59.077663 31826 tablet.cc:2369] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:08:59.112360 31304 tablet.cc:2369] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:59.153438 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 31958
I20240501 20:08:59.270391 31955 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
I20240501 20:08:59.374593 32118 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:59.421299 32118 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:59.427875 31715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.71:35749: connect: Connection refused (error 111)
I20240501 20:08:59.429040 32118 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:08:59.430447 31715 consensus_peers.cc:485] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.71:35749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:02.223673 31715 consensus_peers.cc:485] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.71:35749: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:09:02.918650 31117 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39828:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:02.930100 31238 tablet_service.cc:1467] Processing CreateTablet for tablet d3b0fefa2f5248f49118f7fb59671b6c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.930691 31238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b0fefa2f5248f49118f7fb59671b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.931054 31237 tablet_service.cc:1467] Processing CreateTablet for tablet abdd2df68c774be1ab6e39d9d5dbf563 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.931515 31237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2df68c774be1ab6e39d9d5dbf563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.936901 31369 tablet_service.cc:1467] Processing CreateTablet for tablet d3b0fefa2f5248f49118f7fb59671b6c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.937394 31369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b0fefa2f5248f49118f7fb59671b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.945116 31629 tablet_service.cc:1467] Processing CreateTablet for tablet d3b0fefa2f5248f49118f7fb59671b6c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.945611 31629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b0fefa2f5248f49118f7fb59671b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.946616 31236 tablet_service.cc:1467] Processing CreateTablet for tablet afb53b647def417c8c9e2378dbeb5dd9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.947077 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb53b647def417c8c9e2378dbeb5dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.948640 32161 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:02.948953 31368 tablet_service.cc:1467] Processing CreateTablet for tablet abdd2df68c774be1ab6e39d9d5dbf563 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.949162 31628 tablet_service.cc:1467] Processing CreateTablet for tablet afb53b647def417c8c9e2378dbeb5dd9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.949414 31368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2df68c774be1ab6e39d9d5dbf563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.949553 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb53b647def417c8c9e2378dbeb5dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.950551 32161 tablet_bootstrap.cc:654] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.953405 32160 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:02.955323 32160 tablet_bootstrap.cc:654] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.958510 31759 tablet_service.cc:1467] Processing CreateTablet for tablet afb53b647def417c8c9e2378dbeb5dd9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.958863 31760 tablet_service.cc:1467] Processing CreateTablet for tablet abdd2df68c774be1ab6e39d9d5dbf563 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:02.958981 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb53b647def417c8c9e2378dbeb5dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.963806 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2df68c774be1ab6e39d9d5dbf563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:02.972229 32163 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:02.975661 32163 tablet_bootstrap.cc:654] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.982594 32161 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:02.982849 32161 ts_tablet_manager.cc:1367] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.000s
I20240501 20:09:02.984014 32161 raft_consensus.cc:348] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:02.984414 32161 raft_consensus.cc:374] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.984549 32161 raft_consensus.cc:729] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.984921 32161 consensus_queue.cc:260] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:02.986552 32163 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:02.986768 32163 ts_tablet_manager.cc:1367] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240501 20:09:02.987941 32163 raft_consensus.cc:348] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:02.988147 32165 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:02.988302 32163 raft_consensus.cc:374] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.988452 32163 raft_consensus.cc:729] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.988824 32163 consensus_queue.cc:260] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:02.990078 32165 tablet_bootstrap.cc:654] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.992964 32161 ts_tablet_manager.cc:1398] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240501 20:09:02.993330 32161 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:02.995188 32161 tablet_bootstrap.cc:654] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:02.995429 32160 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:02.995625 32160 ts_tablet_manager.cc:1367] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.013s
I20240501 20:09:02.996681 32160 raft_consensus.cc:348] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:02.996963 32160 raft_consensus.cc:374] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:02.997059 32160 raft_consensus.cc:729] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:02.997323 32160 consensus_queue.cc:260] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:02.997819 32163 ts_tablet_manager.cc:1398] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:09:02.998167 32163 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:03.000869 32160 ts_tablet_manager.cc:1398] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:03.001180 32160 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:03.002038 32163 tablet_bootstrap.cc:654] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.003890 32165 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:03.004124 32165 ts_tablet_manager.cc:1367] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240501 20:09:03.004037 31497 tablet_service.cc:1467] Processing CreateTablet for tablet afb53b647def417c8c9e2378dbeb5dd9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:03.005174 32165 raft_consensus.cc:348] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.005390 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb53b647def417c8c9e2378dbeb5dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:03.005425 32165 raft_consensus.cc:374] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.005518 32165 raft_consensus.cc:729] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.005882 32160 tablet_bootstrap.cc:654] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.005805 32165 consensus_queue.cc:260] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.006453 32165 ts_tablet_manager.cc:1398] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:03.006793 32165 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:03.003931 31499 tablet_service.cc:1467] Processing CreateTablet for tablet d3b0fefa2f5248f49118f7fb59671b6c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:03.007750 31499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b0fefa2f5248f49118f7fb59671b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:03.008677 32165 tablet_bootstrap.cc:654] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.003795 31498 tablet_service.cc:1467] Processing CreateTablet for tablet abdd2df68c774be1ab6e39d9d5dbf563 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:03.009645 31498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2df68c774be1ab6e39d9d5dbf563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:03.010967 32163 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:03.012892 32165 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:03.013283 31888 tablet_service.cc:1467] Processing CreateTablet for tablet afb53b647def417c8c9e2378dbeb5dd9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:03.014358 32161 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:03.014552 32161 ts_tablet_manager.cc:1367] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240501 20:09:03.014731 31888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb53b647def417c8c9e2378dbeb5dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:03.015515 32161 raft_consensus.cc:348] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.015822 32161 raft_consensus.cc:374] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.015895 32161 raft_consensus.cc:729] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.016278 32161 consensus_queue.cc:260] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.017063 32161 ts_tablet_manager.cc:1398] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:03.017120 31890 tablet_service.cc:1467] Processing CreateTablet for tablet d3b0fefa2f5248f49118f7fb59671b6c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:03.017431 32165 ts_tablet_manager.cc:1367] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 20:09:03.017576 31890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b0fefa2f5248f49118f7fb59671b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:03.018256 32165 raft_consensus.cc:348] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.018573 32165 raft_consensus.cc:374] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.018646 32165 raft_consensus.cc:729] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.018918 32165 consensus_queue.cc:260] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.019543 32165 ts_tablet_manager.cc:1398] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:03.013582 32163 ts_tablet_manager.cc:1367] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20240501 20:09:03.020655 32163 raft_consensus.cc:348] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.021019 32163 raft_consensus.cc:374] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.021097 32163 raft_consensus.cc:729] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.021448 32163 consensus_queue.cc:260] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.022228 32163 ts_tablet_manager.cc:1398] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:03.023619 32160 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:03.023847 32160 ts_tablet_manager.cc:1367] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.001s
I20240501 20:09:03.024711 32160 raft_consensus.cc:348] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.025075 32160 raft_consensus.cc:374] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.025177 32160 raft_consensus.cc:729] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.025463 32160 consensus_queue.cc:260] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.026098 32160 ts_tablet_manager.cc:1398] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:03.026450 32160 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:03.026698 31889 tablet_service.cc:1467] Processing CreateTablet for tablet abdd2df68c774be1ab6e39d9d5dbf563 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:03.027303 31889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2df68c774be1ab6e39d9d5dbf563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:03.028132 32160 tablet_bootstrap.cc:654] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.032999 32118 raft_consensus.cc:482] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:03.033190 32118 raft_consensus.cc:504] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.034335 32118 leader_election.cc:288] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:03.036912 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:03.037084 32168 raft_consensus.cc:482] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:03.037259 32168 raft_consensus.cc:504] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
W20240501 20:09:03.038306 31715 leader_election.cc:341] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.039556 32168 leader_election.cc:288] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:03.058691 32184 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:03.057335 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
W20240501 20:09:03.061728 31713 leader_election.cc:341] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.067162 32184 tablet_bootstrap.cc:654] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.068620 32184 log.cc:824] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:03.071825 32175 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:03.075872 32175 tablet_bootstrap.cc:654] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.077281 32175 log.cc:824] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:03.091141 32175 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:03.094055 32175 ts_tablet_manager.cc:1367] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.004s
I20240501 20:09:03.115458 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
W20240501 20:09:03.122476 31712 leader_election.cc:341] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.122740 31712 leader_election.cc:302] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: d73d71e1531b4649b363236998019221; no voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ec65c12cd0724cf9aa16fcd61bbedae4
I20240501 20:09:03.123131 32118 raft_consensus.cc:2726] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:03.128540 32167 raft_consensus.cc:482] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:03.128762 32167 raft_consensus.cc:504] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:03.131255 32167 leader_election.cc:288] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:03.153654 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
W20240501 20:09:03.154520 31322 leader_election.cc:341] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.156661 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
W20240501 20:09:03.157333 31323 leader_election.cc:341] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.160495 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
W20240501 20:09:03.161621 31583 leader_election.cc:341] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.161857 32160 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:03.161824 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:03.162047 32160 ts_tablet_manager.cc:1367] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.136s user 0.032s sys 0.065s
I20240501 20:09:03.162061 31649 raft_consensus.cc:2454] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:09:03.162823 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:03.162899 32160 raft_consensus.cc:348] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.163100 31257 raft_consensus.cc:2454] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:03.163213 32160 raft_consensus.cc:374] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.163406 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:03.164170 32160 raft_consensus.cc:729] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.164217 31258 raft_consensus.cc:2454] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:03.164439 32160 consensus_queue.cc:260] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.164991 32160 ts_tablet_manager.cc:1398] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:03.165648 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
W20240501 20:09:03.166301 31582 leader_election.cc:341] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.170897 32175 raft_consensus.cc:348] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.171327 32175 raft_consensus.cc:374] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.171430 32175 raft_consensus.cc:729] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.172204 32175 consensus_queue.cc:260] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.173967 32184 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:03.174273 32184 ts_tablet_manager.cc:1367] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.116s user 0.012s sys 0.010s
I20240501 20:09:03.175769 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:03.176030 31389 raft_consensus.cc:2454] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:03.176606 31585 leader_election.cc:302] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986; no voters: 582c7a51c0ac43b4934078ebdf434485, ec65c12cd0724cf9aa16fcd61bbedae4
I20240501 20:09:03.177026 32168 raft_consensus.cc:2781] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:03.177125 32168 raft_consensus.cc:482] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:03.177193 32168 raft_consensus.cc:3037] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.179435 32175 ts_tablet_manager.cc:1398] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.084s user 0.009s sys 0.000s
I20240501 20:09:03.179934 32175 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:03.179922 32168 raft_consensus.cc:504] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:03.181298 32168 leader_election.cc:288] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Requested vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:03.182106 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:03.182257 32175 tablet_bootstrap.cc:654] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:03.183949 31583 leader_election.cc:341] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.184227 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
W20240501 20:09:03.185067 31582 leader_election.cc:341] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:03.185053 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:03.185266 31258 raft_consensus.cc:3037] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.188277 31258 raft_consensus.cc:2454] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:03.189064 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:03.189311 31389 raft_consensus.cc:3037] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.192283 31389 raft_consensus.cc:2454] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:03.192912 31585 leader_election.cc:302] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986; no voters: 582c7a51c0ac43b4934078ebdf434485, ec65c12cd0724cf9aa16fcd61bbedae4
I20240501 20:09:03.193514 32168 raft_consensus.cc:2781] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:03.193987 32168 raft_consensus.cc:686] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Becoming Leader. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Running, Role: LEADER
I20240501 20:09:03.194723 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:03.194607 32168 consensus_queue.cc:237] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:03.195343 31322 leader_election.cc:302] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, db07c3a323eb4a3c919eaf9b19b24986; no voters: 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1, ec65c12cd0724cf9aa16fcd61bbedae4
I20240501 20:09:03.195804 32167 raft_consensus.cc:2726] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:03.196715 32175 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:03.196920 32175 ts_tablet_manager.cc:1367] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20240501 20:09:03.197839 32175 raft_consensus.cc:348] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:03.198091 32175 raft_consensus.cc:374] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.198165 32175 raft_consensus.cc:729] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.198458 32175 consensus_queue.cc:260] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:03.199132 32175 ts_tablet_manager.cc:1398] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:03.199404 32175 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:03.199304 31117 catalog_manager.cc:5484] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:03.201666 32175 tablet_bootstrap.cc:654] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.207983 32194 raft_consensus.cc:482] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:03.208226 32194 raft_consensus.cc:504] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.209540 32175 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:03.209748 32175 ts_tablet_manager.cc:1367] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240501 20:09:03.210485 32175 raft_consensus.cc:348] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.210685 32194 leader_election.cc:288] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527)
I20240501 20:09:03.210758 32175 raft_consensus.cc:374] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.210863 32175 raft_consensus.cc:729] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.211102 32175 consensus_queue.cc:260] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.211652 32175 ts_tablet_manager.cc:1398] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:03.216253 32184 raft_consensus.cc:348] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.216710 32184 raft_consensus.cc:374] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.216883 32184 raft_consensus.cc:729] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.217658 32184 consensus_queue.cc:260] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
W20240501 20:09:03.235332 31565 tablet.cc:2369] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:03.242640 32184 ts_tablet_manager.cc:1398] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.068s user 0.016s sys 0.000s
I20240501 20:09:03.243108 32184 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:03.244854 32184 tablet_bootstrap.cc:654] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.247804 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2df68c774be1ab6e39d9d5dbf563" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:03.248221 31910 raft_consensus.cc:2454] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:03.253796 32184 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:03.254019 32184 ts_tablet_manager.cc:1367] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240501 20:09:03.254902 32184 raft_consensus.cc:348] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.254992 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2df68c774be1ab6e39d9d5dbf563" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:03.255146 32184 raft_consensus.cc:374] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.255230 32184 raft_consensus.cc:729] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.255369 31780 raft_consensus.cc:2454] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:03.255491 32184 consensus_queue.cc:260] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.255887 31453 leader_election.cc:302] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, d73d71e1531b4649b363236998019221, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:03.256136 32184 ts_tablet_manager.cc:1398] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:03.256338 32194 raft_consensus.cc:2781] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:03.256487 32194 raft_consensus.cc:482] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:03.256587 32194 raft_consensus.cc:3037] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.256933 32184 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
W20240501 20:09:03.257421 31956 tablet.cc:2369] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:03.258622 32184 tablet_bootstrap.cc:654] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:03.259024 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2df68c774be1ab6e39d9d5dbf563" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:03.259311 31258 raft_consensus.cc:2454] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:03.259398 32194 raft_consensus.cc:504] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.260145 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2df68c774be1ab6e39d9d5dbf563" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:03.260394 31389 raft_consensus.cc:2454] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:03.261564 32194 leader_election.cc:288] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 election: Requested vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527)
I20240501 20:09:03.261590 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2df68c774be1ab6e39d9d5dbf563" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:03.261765 31780 raft_consensus.cc:3037] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.262023 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2df68c774be1ab6e39d9d5dbf563" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:03.262248 31258 raft_consensus.cc:3037] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.263036 32184 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:03.263273 32184 ts_tablet_manager.cc:1367] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 20:09:03.264124 32184 raft_consensus.cc:348] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:03.264485 32184 raft_consensus.cc:374] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:03.264612 32184 raft_consensus.cc:729] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:03.264644 31780 raft_consensus.cc:2454] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:03.264942 31258 raft_consensus.cc:2454] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:03.265290 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2df68c774be1ab6e39d9d5dbf563" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:03.265501 31910 raft_consensus.cc:3037] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.265388 32184 consensus_queue.cc:260] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:03.266228 32184 ts_tablet_manager.cc:1398] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:03.267903 31910 raft_consensus.cc:2454] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:03.268313 31452 leader_election.cc:302] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, d73d71e1531b4649b363236998019221, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:03.268617 32194 raft_consensus.cc:2781] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:03.269083 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdd2df68c774be1ab6e39d9d5dbf563" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:03.269284 31389 raft_consensus.cc:3037] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.271660 31389 raft_consensus.cc:2454] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:03.271916 32194 raft_consensus.cc:686] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Becoming Leader. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Running, Role: LEADER
I20240501 20:09:03.272372 32194 consensus_queue.cc:237] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:03.277348 31117 catalog_manager.cc:5484] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67). New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:03.420771 32171 raft_consensus.cc:482] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:03.420998 32171 raft_consensus.cc:504] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.423202 32171 leader_election.cc:288] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:09:03.435564 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:03.435967 31910 raft_consensus.cc:2454] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:03.439870 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:03.440192 31519 raft_consensus.cc:2454] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:03.440658 31192 leader_election.cc:302] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:03.441219 32171 raft_consensus.cc:2781] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:03.441350 32171 raft_consensus.cc:482] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:03.441454 32171 raft_consensus.cc:3037] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.444456 32171 raft_consensus.cc:504] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.445459 32171 leader_election.cc:288] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 election: Requested vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:09:03.445887 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:03.446138 31910 raft_consensus.cc:3037] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.446231 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:03.446429 31519 raft_consensus.cc:3037] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.447206 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:03.447445 31649 raft_consensus.cc:2454] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:03.447856 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:03.448132 31648 raft_consensus.cc:3037] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.449090 31910 raft_consensus.cc:2454] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 1.
I20240501 20:09:03.449159 31519 raft_consensus.cc:2454] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 1.
I20240501 20:09:03.449906 31192 leader_election.cc:302] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:03.450250 32171 raft_consensus.cc:2781] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:03.450389 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:03.450460 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:03.450663 31780 raft_consensus.cc:2454] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:03.451138 31648 raft_consensus.cc:2454] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 1.
I20240501 20:09:03.452474 32171 raft_consensus.cc:686] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Becoming Leader. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Running, Role: LEADER
I20240501 20:09:03.453022 32171 consensus_queue.cc:237] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:03.457216 31117 catalog_manager.cc:5484] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: term changed from 0 to 1, leader changed from <none> to 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65). New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:03.479110 32203 raft_consensus.cc:482] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:03.479322 32203 raft_consensus.cc:504] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:03.481379 32203 leader_election.cc:288] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:03.491333 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:03.491319 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:03.491623 31519 raft_consensus.cc:2454] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 0.
I20240501 20:09:03.491667 31258 raft_consensus.cc:2379] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 582c7a51c0ac43b4934078ebdf434485 in current term 1: Already voted for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in this term.
I20240501 20:09:03.491750 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:03.495620 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b0fefa2f5248f49118f7fb59671b6c" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:03.495918 31389 raft_consensus.cc:2379] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 582c7a51c0ac43b4934078ebdf434485 in current term 1: Already voted for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in this term.
I20240501 20:09:03.496289 31846 leader_election.cc:302] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, ec65c12cd0724cf9aa16fcd61bbedae4; no voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986
I20240501 20:09:03.496694 32203 raft_consensus.cc:3037] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.499029 32203 raft_consensus.cc:2726] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:03.648092 32168 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.657727 32194 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.772028 32194 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.777357 32194 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.779186 32194 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.814747 31519 raft_consensus.cc:3037] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.817476 32168 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.818444 32168 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.824366 32168 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.874604 32171 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:03.879103 32217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:03.879392 32217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:03.886857 32217 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:03.896598 32171 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.901014 31779 raft_consensus.cc:3037] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:03.901552 32171 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:03.904593 32171 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 328.268 ms
time per row: 0.164134 ms
I20240501 20:09:04.361892 31112 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39828:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:04.370929 31628 tablet_service.cc:1467] Processing CreateTablet for tablet e93ddb68383c43089544a6f0549c83e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.371407 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93ddb68383c43089544a6f0549c83e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.371802 31498 tablet_service.cc:1467] Processing CreateTablet for tablet e93ddb68383c43089544a6f0549c83e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.372231 31498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93ddb68383c43089544a6f0549c83e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.372611 31497 tablet_service.cc:1467] Processing CreateTablet for tablet 791393d238c3413d9472eecffbf1438a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.373126 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791393d238c3413d9472eecffbf1438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.373055 31368 tablet_service.cc:1467] Processing CreateTablet for tablet e93ddb68383c43089544a6f0549c83e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.373461 31368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93ddb68383c43089544a6f0549c83e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.377187 31890 tablet_service.cc:1467] Processing CreateTablet for tablet e93ddb68383c43089544a6f0549c83e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.377607 31890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93ddb68383c43089544a6f0549c83e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.377543 31759 tablet_service.cc:1467] Processing CreateTablet for tablet e93ddb68383c43089544a6f0549c83e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.377938 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93ddb68383c43089544a6f0549c83e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.381114 31760 tablet_service.cc:1467] Processing CreateTablet for tablet 791393d238c3413d9472eecffbf1438a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.381341 31629 tablet_service.cc:1467] Processing CreateTablet for tablet 791393d238c3413d9472eecffbf1438a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.381486 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791393d238c3413d9472eecffbf1438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.381736 31629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791393d238c3413d9472eecffbf1438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.385325 31236 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0a8894c0f46c683461a1857ff27ff (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.385747 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0a8894c0f46c683461a1857ff27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.386368 31369 tablet_service.cc:1467] Processing CreateTablet for tablet 791393d238c3413d9472eecffbf1438a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.386783 31369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791393d238c3413d9472eecffbf1438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.389137 31889 tablet_service.cc:1467] Processing CreateTablet for tablet 791393d238c3413d9472eecffbf1438a (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.389525 31889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791393d238c3413d9472eecffbf1438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.393836 31367 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0a8894c0f46c683461a1857ff27ff (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.394269 31367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0a8894c0f46c683461a1857ff27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.397754 32313 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:04.399612 32313 tablet_bootstrap.cc:654] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.404014 32310 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:04.405674 32310 tablet_bootstrap.cc:654] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.407462 32314 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:04.409475 32314 tablet_bootstrap.cc:654] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.415096 31629 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0a8894c0f46c683461a1857ff27ff (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.415273 31497 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0a8894c0f46c683461a1857ff27ff (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.415544 31629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0a8894c0f46c683461a1857ff27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.415680 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0a8894c0f46c683461a1857ff27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.425606 32315 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:04.426738 32311 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:04.427454 32315 tablet_bootstrap.cc:654] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.428665 32311 tablet_bootstrap.cc:654] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.431113 32314 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:04.431275 32314 ts_tablet_manager.cc:1367] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
I20240501 20:09:04.432049 32314 raft_consensus.cc:348] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.432286 32314 raft_consensus.cc:374] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.432353 32314 raft_consensus.cc:729] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.432600 32314 consensus_queue.cc:260] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.433804 32310 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:04.433960 32310 ts_tablet_manager.cc:1367] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.003s
I20240501 20:09:04.433919 31760 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0a8894c0f46c683461a1857ff27ff (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:04.434662 32310 raft_consensus.cc:348] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.434895 32310 raft_consensus.cc:374] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.434973 32310 raft_consensus.cc:729] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.435210 32310 consensus_queue.cc:260] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.436436 32310 ts_tablet_manager.cc:1398] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:04.437525 32309 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:04.437935 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0a8894c0f46c683461a1857ff27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:04.439484 32309 tablet_bootstrap.cc:654] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.441668 32313 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:04.441849 32313 ts_tablet_manager.cc:1367] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.009s
I20240501 20:09:04.441996 32311 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:04.442157 32311 ts_tablet_manager.cc:1367] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 20:09:04.442627 32313 raft_consensus.cc:348] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.442863 32313 raft_consensus.cc:374] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.442942 32313 raft_consensus.cc:729] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.442993 32311 raft_consensus.cc:348] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.443234 32311 raft_consensus.cc:374] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.443181 32313 consensus_queue.cc:260] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.443331 32311 raft_consensus.cc:729] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.443584 32311 consensus_queue.cc:260] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.444176 32311 ts_tablet_manager.cc:1398] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:04.444440 32311 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:04.446012 32311 tablet_bootstrap.cc:654] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.447207 32314 ts_tablet_manager.cc:1398] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.016s user 0.002s sys 0.003s
I20240501 20:09:04.447516 32314 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:04.447793 32309 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:04.447952 32309 ts_tablet_manager.cc:1367] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:09:04.449115 32314 tablet_bootstrap.cc:654] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.449739 32315 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:04.449901 32315 ts_tablet_manager.cc:1367] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20240501 20:09:04.450713 32315 raft_consensus.cc:348] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.450975 32315 raft_consensus.cc:374] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.451068 32315 raft_consensus.cc:729] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.451336 32315 consensus_queue.cc:260] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.452674 32313 ts_tablet_manager.cc:1398] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.011s user 0.001s sys 0.007s
I20240501 20:09:04.453038 32313 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:04.453833 32315 ts_tablet_manager.cc:1398] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:04.454162 32315 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:04.454967 32313 tablet_bootstrap.cc:654] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.459441 32313 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:04.459606 32313 ts_tablet_manager.cc:1367] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240501 20:09:04.460250 32313 raft_consensus.cc:348] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.460492 32313 raft_consensus.cc:374] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.460566 32313 raft_consensus.cc:729] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.460844 32313 consensus_queue.cc:260] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.460996 32315 tablet_bootstrap.cc:654] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.461442 32313 ts_tablet_manager.cc:1398] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:04.461699 32313 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:04.462677 32309 raft_consensus.cc:348] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.462922 32309 raft_consensus.cc:374] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.463003 32309 raft_consensus.cc:729] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.463241 32313 tablet_bootstrap.cc:654] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.463318 32309 consensus_queue.cc:260] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.464033 32309 ts_tablet_manager.cc:1398] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240501 20:09:04.464358 32309 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:04.467420 32309 tablet_bootstrap.cc:654] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.469518 32315 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:04.469693 32315 ts_tablet_manager.cc:1367] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20240501 20:09:04.470445 32315 raft_consensus.cc:348] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.470708 32315 raft_consensus.cc:374] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.470775 32315 raft_consensus.cc:729] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.471002 32315 consensus_queue.cc:260] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.471562 32315 ts_tablet_manager.cc:1398] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:04.471909 32315 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:04.473721 32315 tablet_bootstrap.cc:654] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.476194 32309 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:04.476441 32309 ts_tablet_manager.cc:1367] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 20:09:04.477339 32309 raft_consensus.cc:348] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.477720 32309 raft_consensus.cc:374] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.477833 32309 raft_consensus.cc:729] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.479243 32315 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:04.479373 32315 ts_tablet_manager.cc:1367] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:09:04.479770 32309 consensus_queue.cc:260] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.480095 32315 raft_consensus.cc:348] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.480465 32315 raft_consensus.cc:374] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.480543 32315 raft_consensus.cc:729] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.481402 32309 ts_tablet_manager.cc:1398] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:04.481768 32309 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:04.480793 32315 consensus_queue.cc:260] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.482954 32315 ts_tablet_manager.cc:1398] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:04.483809 32311 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:04.483990 32311 ts_tablet_manager.cc:1367] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.005s
I20240501 20:09:04.484850 32311 raft_consensus.cc:348] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.485719 32311 raft_consensus.cc:374] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.485805 32311 raft_consensus.cc:729] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.486004 32313 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:04.486161 32313 ts_tablet_manager.cc:1367] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.003s
I20240501 20:09:04.486079 32311 consensus_queue.cc:260] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.486340 32314 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:04.486505 32314 ts_tablet_manager.cc:1367] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.001s
I20240501 20:09:04.486929 32313 raft_consensus.cc:348] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.487246 32313 raft_consensus.cc:374] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.487360 32313 raft_consensus.cc:729] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.487301 32314 raft_consensus.cc:348] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.487535 32314 raft_consensus.cc:374] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.487607 32314 raft_consensus.cc:729] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.487645 32313 consensus_queue.cc:260] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.487882 32314 consensus_queue.cc:260] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.488605 32311 ts_tablet_manager.cc:1398] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:04.488801 32313 ts_tablet_manager.cc:1398] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:04.488898 32311 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:04.489032 32314 ts_tablet_manager.cc:1398] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:04.491734 32311 tablet_bootstrap.cc:654] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.494321 32309 tablet_bootstrap.cc:654] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:04.496057 32311 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:04.496182 32309 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:04.496201 32311 ts_tablet_manager.cc:1367] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240501 20:09:04.496351 32309 ts_tablet_manager.cc:1367] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240501 20:09:04.496837 32311 raft_consensus.cc:348] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.497049 32311 raft_consensus.cc:374] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.497133 32311 raft_consensus.cc:729] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.497176 32309 raft_consensus.cc:348] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.497462 32309 raft_consensus.cc:374] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:04.497355 32311 consensus_queue.cc:260] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.497869 32311 ts_tablet_manager.cc:1398] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:04.497565 32309 raft_consensus.cc:729] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:04.498801 32309 consensus_queue.cc:260] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.499575 32309 ts_tablet_manager.cc:1398] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:04.510455 32168 raft_consensus.cc:482] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:04.510695 32168 raft_consensus.cc:504] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.511708 32168 leader_election.cc:288] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:04.512104 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93ddb68383c43089544a6f0549c83e2" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:04.512148 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93ddb68383c43089544a6f0549c83e2" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:04.512367 31909 raft_consensus.cc:2454] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:04.512394 31389 raft_consensus.cc:2454] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:04.512969 31585 leader_election.cc:302] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:09:04.513146 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93ddb68383c43089544a6f0549c83e2" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:04.513372 32168 raft_consensus.cc:2781] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:04.513397 31518 raft_consensus.cc:2454] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:04.513466 32168 raft_consensus.cc:482] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:04.513563 32168 raft_consensus.cc:3037] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.516155 32168 raft_consensus.cc:504] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.517252 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93ddb68383c43089544a6f0549c83e2" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:04.517473 31389 raft_consensus.cc:3037] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.517483 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93ddb68383c43089544a6f0549c83e2" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:04.517675 31909 raft_consensus.cc:3037] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.517654 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93ddb68383c43089544a6f0549c83e2" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:04.517845 31518 raft_consensus.cc:3037] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.520118 31389 raft_consensus.cc:2454] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:04.520424 31518 raft_consensus.cc:2454] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:04.520423 31909 raft_consensus.cc:2454] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:04.520624 32168 leader_election.cc:288] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Requested vote from peers db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:04.520859 31583 leader_election.cc:302] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:04.520835 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93ddb68383c43089544a6f0549c83e2" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:04.521060 31779 raft_consensus.cc:2454] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:04.521194 32236 raft_consensus.cc:2781] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:04.521346 32236 raft_consensus.cc:686] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Becoming Leader. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Running, Role: LEADER
I20240501 20:09:04.521322 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93ddb68383c43089544a6f0549c83e2" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:04.521505 31780 raft_consensus.cc:3037] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.521739 32236 consensus_queue.cc:237] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:04.523960 31780 raft_consensus.cc:2454] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:04.525313 31116 catalog_manager.cc:5484] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:04.637262 32279 raft_consensus.cc:482] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:04.637477 32279 raft_consensus.cc:504] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.638401 32322 raft_consensus.cc:482] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:04.638450 32279 leader_election.cc:288] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:04.638589 32322 raft_consensus.cc:504] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.639515 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:04.639640 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:04.639772 31780 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:04.639803 32322 leader_election.cc:288] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:04.639864 31649 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:04.640393 31191 leader_election.cc:302] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, d73d71e1531b4649b363236998019221; no voters:
I20240501 20:09:04.640700 32279 raft_consensus.cc:2781] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:04.640856 32279 raft_consensus.cc:482] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:04.640815 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:04.640941 32279 raft_consensus.cc:3037] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.641057 31518 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:04.641898 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791393d238c3413d9472eecffbf1438a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:04.642083 31649 raft_consensus.cc:2454] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 0.
I20240501 20:09:04.642748 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791393d238c3413d9472eecffbf1438a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:04.642943 31518 raft_consensus.cc:2454] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 0.
I20240501 20:09:04.643384 31844 leader_election.cc:302] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:04.643741 32322 raft_consensus.cc:2781] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:04.643853 32322 raft_consensus.cc:482] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:04.643934 32322 raft_consensus.cc:3037] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.643998 32279 raft_consensus.cc:504] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.645709 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791393d238c3413d9472eecffbf1438a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:04.646076 31389 raft_consensus.cc:2454] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 0.
I20240501 20:09:04.646790 32322 raft_consensus.cc:504] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.647415 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:04.647647 31780 raft_consensus.cc:3037] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.648087 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:04.648279 31518 raft_consensus.cc:3037] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.650450 32279 leader_election.cc:288] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 election: Requested vote from peers db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:04.650545 31780 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 1.
I20240501 20:09:04.650784 31518 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 1.
I20240501 20:09:04.651188 31192 leader_election.cc:302] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, d73d71e1531b4649b363236998019221, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:04.651289 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:04.651409 32322 leader_election.cc:288] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 election: Requested vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:04.651469 31649 raft_consensus.cc:3037] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.651703 32279 raft_consensus.cc:2781] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:04.651839 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791393d238c3413d9472eecffbf1438a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:04.651926 32279 raft_consensus.cc:686] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Becoming Leader. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Running, Role: LEADER
I20240501 20:09:04.651842 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791393d238c3413d9472eecffbf1438a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:04.652036 31779 raft_consensus.cc:3037] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.652186 31389 raft_consensus.cc:3037] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.652249 32279 consensus_queue.cc:237] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:04.654176 31649 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 1.
I20240501 20:09:04.654702 31779 raft_consensus.cc:2454] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1.
I20240501 20:09:04.655205 31389 raft_consensus.cc:2454] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1.
I20240501 20:09:04.655550 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791393d238c3413d9472eecffbf1438a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:04.655748 31518 raft_consensus.cc:3037] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.656019 31844 leader_election.cc:302] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, d73d71e1531b4649b363236998019221, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:09:04.656056 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791393d238c3413d9472eecffbf1438a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:04.656344 32322 raft_consensus.cc:2781] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:04.656383 31780 raft_consensus.cc:2362] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1. Re-sending same reply.
I20240501 20:09:04.656560 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "791393d238c3413d9472eecffbf1438a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:04.656795 31649 raft_consensus.cc:3037] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.657845 32322 raft_consensus.cc:686] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Becoming Leader. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Running, Role: LEADER
I20240501 20:09:04.658421 32322 consensus_queue.cc:237] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:04.658849 31518 raft_consensus.cc:2454] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1.
I20240501 20:09:04.659263 31649 raft_consensus.cc:2454] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1.
I20240501 20:09:04.661180 31116 catalog_manager.cc:5484] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: term changed from 0 to 1, leader changed from <none> to 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65). New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:04.661998 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:04.662216 31389 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:04.662537 31388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:04.662811 31388 raft_consensus.cc:3037] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:04.663538 31110 catalog_manager.cc:5484] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70). New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:04.665220 31388 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 1.
W20240501 20:09:04.868881 31715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.71:35749: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:09:04.869918 31715 consensus_peers.cc:485] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.71:35749: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:09:04.907236 32236 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:04.981823 32236 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.007090 32236 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.007522 32236 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:05.042116 32343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:05.042434 32343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:09:05.045279 32279 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:05.050951 32343 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:05.052857 32322 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 20:09:05.108788 32279 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.131229 32278 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.151006 32278 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.155450 32322 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.176100 32322 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:05.212921 32389 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 390.915 ms
time per row: 0.195458 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:06.457717 31117 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39828:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:06.466900 31890 tablet_service.cc:1467] Processing CreateTablet for tablet dfe38cf1f2ef48e4adca9910e82e2d9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.467398 31890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe38cf1f2ef48e4adca9910e82e2d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.467675 31236 tablet_service.cc:1467] Processing CreateTablet for tablet dfe38cf1f2ef48e4adca9910e82e2d9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.468086 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe38cf1f2ef48e4adca9910e82e2d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.470832 31368 tablet_service.cc:1467] Processing CreateTablet for tablet dfe38cf1f2ef48e4adca9910e82e2d9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.471243 31368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe38cf1f2ef48e4adca9910e82e2d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.473161 31498 tablet_service.cc:1467] Processing CreateTablet for tablet dfe38cf1f2ef48e4adca9910e82e2d9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.473565 31498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe38cf1f2ef48e4adca9910e82e2d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.477160 32433 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:06.477777 31760 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9d9115e89c437ca7b962201115f087 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.478049 31629 tablet_service.cc:1467] Processing CreateTablet for tablet dda27412e67c4571bd8c5e86a65b787f (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.478199 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9d9115e89c437ca7b962201115f087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.478195 31497 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9d9115e89c437ca7b962201115f087 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.478451 31629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda27412e67c4571bd8c5e86a65b787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.478554 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9d9115e89c437ca7b962201115f087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.481105 31759 tablet_service.cc:1467] Processing CreateTablet for tablet dda27412e67c4571bd8c5e86a65b787f (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.481460 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda27412e67c4571bd8c5e86a65b787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.481665 31367 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9d9115e89c437ca7b962201115f087 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.481807 31238 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9d9115e89c437ca7b962201115f087 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.482039 31367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9d9115e89c437ca7b962201115f087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.482163 31238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9d9115e89c437ca7b962201115f087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.485075 31237 tablet_service.cc:1467] Processing CreateTablet for tablet dda27412e67c4571bd8c5e86a65b787f (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.485105 31628 tablet_service.cc:1467] Processing CreateTablet for tablet dfe38cf1f2ef48e4adca9910e82e2d9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.485420 31237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda27412e67c4571bd8c5e86a65b787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.485464 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe38cf1f2ef48e4adca9910e82e2d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.488324 32433 tablet_bootstrap.cc:654] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.488343 31369 tablet_service.cc:1467] Processing CreateTablet for tablet dda27412e67c4571bd8c5e86a65b787f (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.488756 31369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda27412e67c4571bd8c5e86a65b787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.490442 32435 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:06.491904 31889 tablet_service.cc:1467] Processing CreateTablet for tablet dda27412e67c4571bd8c5e86a65b787f (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.492367 31889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda27412e67c4571bd8c5e86a65b787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.493680 32435 tablet_bootstrap.cc:654] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.500231 32435 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:06.500402 32435 ts_tablet_manager.cc:1367] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240501 20:09:06.501189 32435 raft_consensus.cc:348] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.503296 32435 raft_consensus.cc:374] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.503597 32435 raft_consensus.cc:729] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.504168 32435 consensus_queue.cc:260] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.505486 32435 ts_tablet_manager.cc:1398] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:06.506428 32435 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:06.508412 32435 tablet_bootstrap.cc:654] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.512156 32433 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:06.512403 32433 ts_tablet_manager.cc:1367] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.003s
I20240501 20:09:06.513679 32433 raft_consensus.cc:348] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.514180 32433 raft_consensus.cc:374] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.514366 32433 raft_consensus.cc:729] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.514233 32434 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:06.514739 32433 consensus_queue.cc:260] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.516016 32433 ts_tablet_manager.cc:1398] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:06.516309 32434 tablet_bootstrap.cc:654] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.522682 32433 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:06.524334 32433 tablet_bootstrap.cc:654] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.524344 32434 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:06.524487 32434 ts_tablet_manager.cc:1367] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240501 20:09:06.525363 32434 raft_consensus.cc:348] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.525686 32434 raft_consensus.cc:374] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.525805 32434 raft_consensus.cc:729] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.526115 32434 consensus_queue.cc:260] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.526890 32434 ts_tablet_manager.cc:1398] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:06.527256 32434 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:06.528977 32434 tablet_bootstrap.cc:654] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.532101 31890 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9d9115e89c437ca7b962201115f087 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:06.532585 31890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c9d9115e89c437ca7b962201115f087. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.539778 32433 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:06.540058 32433 ts_tablet_manager.cc:1367] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20240501 20:09:06.540966 32433 raft_consensus.cc:348] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.541334 32433 raft_consensus.cc:374] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.541451 32433 raft_consensus.cc:729] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.541782 32433 consensus_queue.cc:260] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.542448 32433 ts_tablet_manager.cc:1398] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:06.542665 32440 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:06.543857 32434 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:06.544101 32434 ts_tablet_manager.cc:1367] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.013s
I20240501 20:09:06.544950 32442 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:06.545013 32434 raft_consensus.cc:348] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.545351 32434 raft_consensus.cc:374] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.545459 32434 raft_consensus.cc:729] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.545765 32434 consensus_queue.cc:260] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.546478 32434 ts_tablet_manager.cc:1398] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:06.546810 32434 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:06.549167 32434 tablet_bootstrap.cc:654] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.549815 32440 tablet_bootstrap.cc:654] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.554685 32442 tablet_bootstrap.cc:654] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.561317 32436 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:06.564242 32440 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:06.564342 32436 tablet_bootstrap.cc:654] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.564494 32440 ts_tablet_manager.cc:1367] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20240501 20:09:06.565418 32434 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:06.565274 32440 raft_consensus.cc:348] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.565505 32440 raft_consensus.cc:374] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.565601 32440 raft_consensus.cc:729] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.565752 32442 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:06.565814 32434 ts_tablet_manager.cc:1367] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20240501 20:09:06.565966 32442 ts_tablet_manager.cc:1367] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 20:09:06.565848 32440 consensus_queue.cc:260] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.566612 32434 raft_consensus.cc:348] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.566828 32433 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:06.566932 32434 raft_consensus.cc:374] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.567087 32434 raft_consensus.cc:729] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.567206 32440 ts_tablet_manager.cc:1398] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:06.567358 32434 consensus_queue.cc:260] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.567999 32434 ts_tablet_manager.cc:1398] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:06.568360 32433 tablet_bootstrap.cc:654] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.566810 32442 raft_consensus.cc:348] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.568758 32442 raft_consensus.cc:374] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.570057 32435 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:06.570333 32442 raft_consensus.cc:729] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.570299 32435 ts_tablet_manager.cc:1367] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.004s
I20240501 20:09:06.570735 32442 consensus_queue.cc:260] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.571763 32236 raft_consensus.cc:482] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:06.571929 32236 raft_consensus.cc:504] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.572890 32433 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:06.572928 32236 leader_election.cc:288] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527)
I20240501 20:09:06.573041 32433 ts_tablet_manager.cc:1367] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240501 20:09:06.573115 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:06.573318 31258 raft_consensus.cc:2454] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:06.573809 32433 raft_consensus.cc:348] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.574074 32433 raft_consensus.cc:374] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.574160 32433 raft_consensus.cc:729] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.574401 32433 consensus_queue.cc:260] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.574544 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:06.574779 31909 raft_consensus.cc:2454] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:06.575762 32433 ts_tablet_manager.cc:1398] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:06.575951 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:06.576910 31582 leader_election.cc:302] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1; no voters:
W20240501 20:09:06.577574 31585 leader_election.cc:341] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:06.578107 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
W20240501 20:09:06.578864 31583 leader_election.cc:341] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d73d71e1531b4649b363236998019221 (127.28.136.69:43505): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:06.581055 32236 raft_consensus.cc:2781] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:06.581147 32236 raft_consensus.cc:482] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:06.581233 32236 raft_consensus.cc:3037] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.582445 32436 tablet_bootstrap.cc:492] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:06.582615 32436 ts_tablet_manager.cc:1367] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20240501 20:09:06.583397 32436 raft_consensus.cc:348] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.583657 32436 raft_consensus.cc:374] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.583739 32436 raft_consensus.cc:729] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.583977 32436 consensus_queue.cc:260] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.585747 32236 raft_consensus.cc:504] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.587033 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:06.587253 31258 raft_consensus.cc:3037] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.589010 32436 ts_tablet_manager.cc:1398] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240501 20:09:06.589332 32436 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:06.590305 31258 raft_consensus.cc:2454] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:06.591055 32436 tablet_bootstrap.cc:654] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.597102 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:06.597384 31909 raft_consensus.cc:3037] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.597491 32236 leader_election.cc:288] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Requested vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527)
I20240501 20:09:06.598027 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:06.598415 32435 raft_consensus.cc:348] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.598692 32435 raft_consensus.cc:374] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 20:09:06.598662 31585 leader_election.cc:341] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:06.598766 32435 raft_consensus.cc:729] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.599020 32435 consensus_queue.cc:260] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.599510 32435 ts_tablet_manager.cc:1398] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.028s user 0.001s sys 0.000s
I20240501 20:09:06.599725 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
W20240501 20:09:06.600301 31583 leader_election.cc:341] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d73d71e1531b4649b363236998019221 (127.28.136.69:43505): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:06.600359 31909 raft_consensus.cc:2454] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:06.600441 32440 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:06.600859 31582 leader_election.cc:302] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1; no voters: d73d71e1531b4649b363236998019221, db07c3a323eb4a3c919eaf9b19b24986
I20240501 20:09:06.601387 32236 raft_consensus.cc:2781] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:06.601545 32236 raft_consensus.cc:686] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Becoming Leader. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Running, Role: LEADER
I20240501 20:09:06.602011 32236 consensus_queue.cc:237] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.602913 32440 tablet_bootstrap.cc:654] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.604279 32442 ts_tablet_manager.cc:1398] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.038s user 0.012s sys 0.019s
I20240501 20:09:06.604583 32442 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:06.606190 32442 tablet_bootstrap.cc:654] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.608106 32440 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:06.608284 32440 ts_tablet_manager.cc:1367] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.002s
I20240501 20:09:06.608470 32436 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:06.608664 32436 ts_tablet_manager.cc:1367] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.001s
I20240501 20:09:06.609000 32442 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:06.609215 32442 ts_tablet_manager.cc:1367] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:06.609086 32440 raft_consensus.cc:348] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.609321 32440 raft_consensus.cc:374] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.609398 32440 raft_consensus.cc:729] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.608978 31113 catalog_manager.cc:5484] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:06.609405 32436 raft_consensus.cc:348] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.609664 32436 raft_consensus.cc:374] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.609741 32436 raft_consensus.cc:729] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.609678 32440 consensus_queue.cc:260] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.610612 32442 raft_consensus.cc:348] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.609974 32436 consensus_queue.cc:260] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.610957 32442 raft_consensus.cc:374] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.611086 32442 raft_consensus.cc:729] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.611393 32436 ts_tablet_manager.cc:1398] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:06.611374 32442 consensus_queue.cc:260] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.611696 32436 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:06.612097 32442 ts_tablet_manager.cc:1398] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:06.610961 32440 ts_tablet_manager.cc:1398] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:06.613404 32436 tablet_bootstrap.cc:654] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:06.617339 32436 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:06.617473 32436 ts_tablet_manager.cc:1367] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20240501 20:09:06.617995 32436 raft_consensus.cc:348] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.618163 32436 raft_consensus.cc:374] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:06.618227 32436 raft_consensus.cc:729] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:06.618425 32436 consensus_queue.cc:260] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.618829 32436 ts_tablet_manager.cc:1398] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:06.692713 32236 raft_consensus.cc:482] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:06.692906 32236 raft_consensus.cc:504] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.693754 32236 leader_election.cc:288] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:06.693928 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:06.694007 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:06.694160 31909 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:06.694226 31258 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:06.694162 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:06.694420 31389 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:06.694418 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:06.694658 31518 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 0.
I20240501 20:09:06.694754 31582 leader_election.cc:302] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1; no voters:
I20240501 20:09:06.695073 32236 raft_consensus.cc:2781] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:06.695233 32236 raft_consensus.cc:482] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:06.695315 32236 raft_consensus.cc:3037] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.697247 32236 raft_consensus.cc:504] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.698020 32236 leader_election.cc:288] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Requested vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:06.698221 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:06.698427 31258 raft_consensus.cc:3037] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.698436 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:06.698707 31389 raft_consensus.cc:3037] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.698751 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:06.698954 31518 raft_consensus.cc:3037] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.699396 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:06.699579 31909 raft_consensus.cc:3037] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.701097 31258 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:06.701409 31389 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:06.701567 31518 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:06.701721 31585 leader_election.cc:302] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:09:06.701984 32236 raft_consensus.cc:2781] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:06.702105 32236 raft_consensus.cc:686] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Becoming Leader. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Running, Role: LEADER
I20240501 20:09:06.702260 31909 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in term 1.
I20240501 20:09:06.702402 32236 consensus_queue.cc:237] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:06.705379 31117 catalog_manager.cc:5484] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:06.717913 32452 raft_consensus.cc:482] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:06.718081 32452 raft_consensus.cc:504] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } }
I20240501 20:09:06.718998 32452 leader_election.cc:288] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:09:06.719451 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:06.719496 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:06.719488 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:06.719755 31258 raft_consensus.cc:2379] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db07c3a323eb4a3c919eaf9b19b24986 in current term 1: Already voted for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in this term.
I20240501 20:09:06.719825 31909 raft_consensus.cc:2379] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db07c3a323eb4a3c919eaf9b19b24986 in current term 1: Already voted for candidate ce9dba79a8c94a7b87bf51e9ed4b5fe1 in this term.
I20240501 20:09:06.720640 31322 leader_election.cc:302] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: db07c3a323eb4a3c919eaf9b19b24986; no voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1
I20240501 20:09:06.720927 32452 raft_consensus.cc:3037] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.723330 32452 raft_consensus.cc:2726] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:06.725771 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda27412e67c4571bd8c5e86a65b787f" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:06.725993 31779 raft_consensus.cc:2454] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:06.891675 32452 raft_consensus.cc:482] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:06.891893 32452 raft_consensus.cc:504] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.892768 32452 leader_election.cc:288] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:09:06.893206 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9d9115e89c437ca7b962201115f087" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:06.893198 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9d9115e89c437ca7b962201115f087" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:06.893257 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9d9115e89c437ca7b962201115f087" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:06.893464 31779 raft_consensus.cc:2454] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:06.893481 31518 raft_consensus.cc:2454] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:06.893473 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9d9115e89c437ca7b962201115f087" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:06.893527 31258 raft_consensus.cc:2454] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:06.893707 31909 raft_consensus.cc:2454] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:06.893988 31323 leader_election.cc:302] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: d73d71e1531b4649b363236998019221, db07c3a323eb4a3c919eaf9b19b24986, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:06.894307 32452 raft_consensus.cc:2781] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:06.894390 32452 raft_consensus.cc:482] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:06.894474 32452 raft_consensus.cc:3037] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.896463 32452 raft_consensus.cc:504] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.897321 32452 leader_election.cc:288] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 election: Requested vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:09:06.897529 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9d9115e89c437ca7b962201115f087" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:06.897696 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9d9115e89c437ca7b962201115f087" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:06.897687 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9d9115e89c437ca7b962201115f087" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:06.897759 31779 raft_consensus.cc:3037] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.897738 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c9d9115e89c437ca7b962201115f087" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:06.897879 31518 raft_consensus.cc:3037] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.897895 31909 raft_consensus.cc:3037] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.898041 31258 raft_consensus.cc:3037] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:06.900425 31779 raft_consensus.cc:2454] T 2c9d9115e89c437ca7b962201115f087 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:06.900511 31909 raft_consensus.cc:2454] T 2c9d9115e89c437ca7b962201115f087 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:06.900527 31518 raft_consensus.cc:2454] T 2c9d9115e89c437ca7b962201115f087 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:06.900785 31258 raft_consensus.cc:2454] T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:06.900945 31323 leader_election.cc:302] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, db07c3a323eb4a3c919eaf9b19b24986, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:06.901237 32452 raft_consensus.cc:2781] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:06.901654 32452 raft_consensus.cc:686] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Becoming Leader. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Running, Role: LEADER
I20240501 20:09:06.902076 32452 consensus_queue.cc:237] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:06.905905 31113 catalog_manager.cc:5484] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: term changed from 0 to 1, leader changed from <none> to db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "db07c3a323eb4a3c919eaf9b19b24986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:07.039724 32236 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.041615 32236 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.050359 32451 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.058938 31779 raft_consensus.cc:3037] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:09:07.079295 31715 consensus_peers.cc:485] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.71:35749: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:09:07.105119 32236 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.168052 32451 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.247514 32451 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.252704 32451 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.256393 32476 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:09:07.284530 32452 consensus_queue.cc:1035] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.299731 32452 consensus_queue.cc:1035] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.309425 32482 consensus_queue.cc:1035] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:07.311376 32482 consensus_queue.cc:1035] T 2c9d9115e89c437ca7b962201115f087 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:07.323638 32463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:07.323956 32463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:07.332973 32463 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
W20240501 20:09:07.592707 31715 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 115102us
INSERT report
rows total: 2000
time total: 382.577 ms
time per row: 0.191288 ms
I20240501 20:09:07.860626 31108 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39828:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:07.869542 31760 tablet_service.cc:1467] Processing CreateTablet for tablet 7faad10e555240378c4deeb23188128a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.869799 31236 tablet_service.cc:1467] Processing CreateTablet for tablet 7faad10e555240378c4deeb23188128a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.870052 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faad10e555240378c4deeb23188128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.870218 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faad10e555240378c4deeb23188128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.872866 31890 tablet_service.cc:1467] Processing CreateTablet for tablet 7faad10e555240378c4deeb23188128a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.872925 31627 tablet_service.cc:1467] Processing CreateTablet for tablet 87de1a5930394b2ca4ed6e99d01936d7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.873114 31629 tablet_service.cc:1467] Processing CreateTablet for tablet 7faad10e555240378c4deeb23188128a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.873302 31890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faad10e555240378c4deeb23188128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.873440 31627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87de1a5930394b2ca4ed6e99d01936d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.873432 31367 tablet_service.cc:1467] Processing CreateTablet for tablet 87de1a5930394b2ca4ed6e99d01936d7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.873826 31367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87de1a5930394b2ca4ed6e99d01936d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.873796 31628 tablet_service.cc:1467] Processing CreateTablet for tablet ef4b7168ca5f4b2ba49ec83e57b48009 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.874138 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b7168ca5f4b2ba49ec83e57b48009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.877133 31889 tablet_service.cc:1467] Processing CreateTablet for tablet ef4b7168ca5f4b2ba49ec83e57b48009 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.877457 31889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b7168ca5f4b2ba49ec83e57b48009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.877593 31497 tablet_service.cc:1467] Processing CreateTablet for tablet ef4b7168ca5f4b2ba49ec83e57b48009 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.877622 31498 tablet_service.cc:1467] Processing CreateTablet for tablet 7faad10e555240378c4deeb23188128a (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.878000 31497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b7168ca5f4b2ba49ec83e57b48009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.878388 31629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faad10e555240378c4deeb23188128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.881003 31498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faad10e555240378c4deeb23188128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.881179 31238 tablet_service.cc:1467] Processing CreateTablet for tablet ef4b7168ca5f4b2ba49ec83e57b48009 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.881574 31238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b7168ca5f4b2ba49ec83e57b48009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.882566 31237 tablet_service.cc:1467] Processing CreateTablet for tablet 87de1a5930394b2ca4ed6e99d01936d7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.882894 31237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87de1a5930394b2ca4ed6e99d01936d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.885347 31499 tablet_service.cc:1467] Processing CreateTablet for tablet 87de1a5930394b2ca4ed6e99d01936d7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.885700 31499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87de1a5930394b2ca4ed6e99d01936d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.889555 31759 tablet_service.cc:1467] Processing CreateTablet for tablet ef4b7168ca5f4b2ba49ec83e57b48009 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.890426 31758 tablet_service.cc:1467] Processing CreateTablet for tablet 87de1a5930394b2ca4ed6e99d01936d7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:07.892371 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b7168ca5f4b2ba49ec83e57b48009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.896968 31758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87de1a5930394b2ca4ed6e99d01936d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:07.898803 32557 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:07.900871 32557 tablet_bootstrap.cc:654] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.906908 32554 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:07.910189 32559 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:07.910512 32554 tablet_bootstrap.cc:654] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.911873 32559 tablet_bootstrap.cc:654] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.920047 32554 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:07.920235 32554 ts_tablet_manager.cc:1367] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240501 20:09:07.920823 32559 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:07.920980 32559 ts_tablet_manager.cc:1367] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:09:07.921090 32554 raft_consensus.cc:348] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.921581 32554 raft_consensus.cc:374] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.921682 32554 raft_consensus.cc:729] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.921741 32559 raft_consensus.cc:348] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.921996 32559 raft_consensus.cc:374] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.922087 32559 raft_consensus.cc:729] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.921941 32554 consensus_queue.cc:260] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.922680 32554 ts_tablet_manager.cc:1398] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:07.924297 32557 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:07.924479 32557 ts_tablet_manager.cc:1367] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.006s
I20240501 20:09:07.924963 32554 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:07.925297 32557 raft_consensus.cc:348] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.925527 32557 raft_consensus.cc:374] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.925617 32557 raft_consensus.cc:729] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.926980 32555 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:07.927904 32557 consensus_queue.cc:260] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.928047 32559 consensus_queue.cc:260] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.927800 32553 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:07.928756 32557 ts_tablet_manager.cc:1398] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:07.928905 32556 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:07.929025 32557 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:07.930022 32559 ts_tablet_manager.cc:1398] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:09:07.930581 32557 tablet_bootstrap.cc:654] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.931416 32555 tablet_bootstrap.cc:654] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.932088 32556 tablet_bootstrap.cc:654] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.932296 32559 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:07.933863 32559 tablet_bootstrap.cc:654] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.934350 32553 tablet_bootstrap.cc:654] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.937857 32554 tablet_bootstrap.cc:654] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.938750 32553 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:07.938961 32553 ts_tablet_manager.cc:1367] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240501 20:09:07.939805 32553 raft_consensus.cc:348] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.940145 32553 raft_consensus.cc:374] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.940286 32553 raft_consensus.cc:729] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.940594 32553 consensus_queue.cc:260] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.941973 32553 ts_tablet_manager.cc:1398] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:07.944231 32554 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:07.944422 32554 ts_tablet_manager.cc:1367] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240501 20:09:07.944802 32559 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:07.944841 32555 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:07.944986 32555 ts_tablet_manager.cc:1367] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20240501 20:09:07.945086 32559 ts_tablet_manager.cc:1367] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 20:09:07.945257 32554 raft_consensus.cc:348] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.945533 32554 raft_consensus.cc:374] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.945622 32554 raft_consensus.cc:729] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.945739 32555 raft_consensus.cc:348] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.945974 32555 raft_consensus.cc:374] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.945914 32554 consensus_queue.cc:260] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.946071 32555 raft_consensus.cc:729] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.946693 32554 ts_tablet_manager.cc:1398] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:07.946659 32555 consensus_queue.cc:260] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.947085 32559 raft_consensus.cc:348] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.947404 32559 raft_consensus.cc:374] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.947516 32559 raft_consensus.cc:729] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.947831 32559 consensus_queue.cc:260] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.948716 32554 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:07.949587 32555 ts_tablet_manager.cc:1398] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:07.949800 32559 ts_tablet_manager.cc:1398] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:07.951450 32555 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:07.953320 32555 tablet_bootstrap.cc:654] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.954303 32554 tablet_bootstrap.cc:654] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.955466 32556 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:07.955643 32556 ts_tablet_manager.cc:1367] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.002s
I20240501 20:09:07.956307 32557 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:07.956526 32557 ts_tablet_manager.cc:1367] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.007s
I20240501 20:09:07.956686 32554 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:07.956477 32556 raft_consensus.cc:348] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.956799 32556 raft_consensus.cc:374] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.956836 32554 ts_tablet_manager.cc:1367] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240501 20:09:07.956897 32556 raft_consensus.cc:729] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.957171 32556 consensus_queue.cc:260] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.957741 32557 raft_consensus.cc:348] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.958041 32557 raft_consensus.cc:374] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.958163 32557 raft_consensus.cc:729] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.958742 32556 ts_tablet_manager.cc:1398] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:07.959045 32556 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:07.957489 32554 raft_consensus.cc:348] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.960247 32554 raft_consensus.cc:374] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.960340 32554 raft_consensus.cc:729] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.960760 32554 consensus_queue.cc:260] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.958647 32557 consensus_queue.cc:260] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.961416 32556 tablet_bootstrap.cc:654] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.962836 32557 ts_tablet_manager.cc:1398] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:09:07.963168 32555 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:07.963171 32557 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:07.963387 32555 ts_tablet_manager.cc:1367] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 20:09:07.964063 32555 raft_consensus.cc:348] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.964313 32555 raft_consensus.cc:374] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.964391 32555 raft_consensus.cc:729] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.964718 32555 consensus_queue.cc:260] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.964838 32554 ts_tablet_manager.cc:1398] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 20:09:07.965394 32555 ts_tablet_manager.cc:1398] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:07.965771 32557 tablet_bootstrap.cc:654] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.965847 32555 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:07.967391 32555 tablet_bootstrap.cc:654] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.970039 32557 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:07.970180 32557 ts_tablet_manager.cc:1367] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 20:09:07.970666 32556 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:07.970847 32556 ts_tablet_manager.cc:1367] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240501 20:09:07.970916 32557 raft_consensus.cc:348] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.971171 32557 raft_consensus.cc:374] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.971267 32557 raft_consensus.cc:729] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.971508 32557 consensus_queue.cc:260] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.971604 32556 raft_consensus.cc:348] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.971904 32556 raft_consensus.cc:374] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.971983 32556 raft_consensus.cc:729] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.972221 32556 consensus_queue.cc:260] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.972844 32556 ts_tablet_manager.cc:1398] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:07.972867 32557 ts_tablet_manager.cc:1398] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:07.973167 32556 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:07.974665 32556 tablet_bootstrap.cc:654] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:07.977191 32556 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:07.977617 32556 ts_tablet_manager.cc:1367] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:07.978083 32555 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:07.978231 32555 ts_tablet_manager.cc:1367] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240501 20:09:07.978276 32556 raft_consensus.cc:348] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.978566 32556 raft_consensus.cc:374] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.978657 32556 raft_consensus.cc:729] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.978873 32556 consensus_queue.cc:260] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:07.978888 32555 raft_consensus.cc:348] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.979421 32556 ts_tablet_manager.cc:1398] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:07.979100 32555 raft_consensus.cc:374] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:07.979815 32555 raft_consensus.cc:729] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:07.980077 32555 consensus_queue.cc:260] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.980815 32555 ts_tablet_manager.cc:1398] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:07.993862 32513 raft_consensus.cc:482] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:07.994032 32513 raft_consensus.cc:504] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.994875 32513 leader_election.cc:288] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:07.995261 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87de1a5930394b2ca4ed6e99d01936d7" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:07.995491 31257 raft_consensus.cc:2454] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:09:07.995527 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87de1a5930394b2ca4ed6e99d01936d7" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:07.995589 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87de1a5930394b2ca4ed6e99d01936d7" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:07.995777 31648 raft_consensus.cc:2454] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:09:07.995791 31519 raft_consensus.cc:2454] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:09:07.996098 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87de1a5930394b2ca4ed6e99d01936d7" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:07.996234 31712 leader_election.cc:302] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 4 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, d73d71e1531b4649b363236998019221, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:07.996305 31389 raft_consensus.cc:2454] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 0.
I20240501 20:09:07.996527 32513 raft_consensus.cc:2781] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:07.996659 32513 raft_consensus.cc:482] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:07.996771 32513 raft_consensus.cc:3037] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.998662 32513 raft_consensus.cc:504] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:07.999672 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87de1a5930394b2ca4ed6e99d01936d7" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:07.999728 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87de1a5930394b2ca4ed6e99d01936d7" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:07.999713 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87de1a5930394b2ca4ed6e99d01936d7" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:07.999852 31257 raft_consensus.cc:3037] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.999898 31519 raft_consensus.cc:3037] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:07.999918 31648 raft_consensus.cc:3037] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.001119 31389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87de1a5930394b2ca4ed6e99d01936d7" candidate_uuid: "d73d71e1531b4649b363236998019221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:08.001325 31389 raft_consensus.cc:3037] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.002599 31648 raft_consensus.cc:2454] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 1.
I20240501 20:09:08.002655 31257 raft_consensus.cc:2454] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 1.
I20240501 20:09:08.003052 32513 leader_election.cc:288] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 election: Requested vote from peers db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:08.003322 31715 leader_election.cc:302] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, d73d71e1531b4649b363236998019221; no voters:
I20240501 20:09:08.003654 32513 raft_consensus.cc:2781] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:08.003782 32513 raft_consensus.cc:686] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 1 LEADER]: Becoming Leader. State: Replica: d73d71e1531b4649b363236998019221, State: Running, Role: LEADER
I20240501 20:09:08.004051 32513 consensus_queue.cc:237] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:08.004308 31389 raft_consensus.cc:2454] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 1.
I20240501 20:09:08.006464 31519 raft_consensus.cc:2454] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73d71e1531b4649b363236998019221 in term 1.
I20240501 20:09:08.007421 31117 catalog_manager.cc:5484] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 reported cstate change: term changed from 0 to 1, leader changed from <none> to d73d71e1531b4649b363236998019221 (127.28.136.69). New cstate: current_term: 1 leader_uuid: "d73d71e1531b4649b363236998019221" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:08.021610 32389 raft_consensus.cc:482] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:08.021829 32389 raft_consensus.cc:504] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:08.022984 32389 leader_election.cc:288] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:08.023142 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faad10e555240378c4deeb23188128a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:08.023415 31258 raft_consensus.cc:2454] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 0.
I20240501 20:09:08.023626 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faad10e555240378c4deeb23188128a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:08.023860 31779 raft_consensus.cc:2454] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 0.
I20240501 20:09:08.024159 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faad10e555240378c4deeb23188128a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:08.023995 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faad10e555240378c4deeb23188128a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:08.024312 31844 leader_election.cc:302] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, d73d71e1531b4649b363236998019221; no voters:
I20240501 20:09:08.024379 31648 raft_consensus.cc:2454] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 0.
I20240501 20:09:08.024398 31519 raft_consensus.cc:2454] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 0.
I20240501 20:09:08.024596 32389 raft_consensus.cc:2781] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:08.024704 32389 raft_consensus.cc:482] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:08.024858 32389 raft_consensus.cc:3037] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.026762 32389 raft_consensus.cc:504] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:08.027608 32389 leader_election.cc:288] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 election: Requested vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:08.027804 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faad10e555240378c4deeb23188128a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:08.028074 31779 raft_consensus.cc:3037] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.027967 31519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faad10e555240378c4deeb23188128a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:08.028054 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faad10e555240378c4deeb23188128a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:08.028254 31519 raft_consensus.cc:3037] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.028264 31257 raft_consensus.cc:3037] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.028215 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7faad10e555240378c4deeb23188128a" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:08.028393 31648 raft_consensus.cc:3037] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.030799 32344 raft_consensus.cc:482] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:08.030975 32344 raft_consensus.cc:504] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:08.031280 31519 raft_consensus.cc:2454] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1.
I20240501 20:09:08.031209 31257 raft_consensus.cc:2454] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1.
I20240501 20:09:08.031287 31648 raft_consensus.cc:2454] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1.
I20240501 20:09:08.031773 31846 leader_election.cc:302] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:08.032058 32389 raft_consensus.cc:2781] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:08.032245 32344 leader_election.cc:288] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:09:08.032965 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b7168ca5f4b2ba49ec83e57b48009" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:08.032958 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b7168ca5f4b2ba49ec83e57b48009" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:08.033180 31257 raft_consensus.cc:2454] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:08.033195 31780 raft_consensus.cc:2454] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:08.033985 31453 leader_election.cc:302] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, d73d71e1531b4649b363236998019221, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:08.034117 31779 raft_consensus.cc:2454] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 582c7a51c0ac43b4934078ebdf434485 in term 1.
I20240501 20:09:08.034283 32344 raft_consensus.cc:2781] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:08.034395 32344 raft_consensus.cc:482] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:08.034487 32344 raft_consensus.cc:3037] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.035105 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b7168ca5f4b2ba49ec83e57b48009" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:08.035331 31910 raft_consensus.cc:2454] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:08.035854 32389 raft_consensus.cc:686] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Becoming Leader. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Running, Role: LEADER
I20240501 20:09:08.036231 32389 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:08.037322 32344 raft_consensus.cc:504] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:08.038527 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b7168ca5f4b2ba49ec83e57b48009" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:08.038745 31779 raft_consensus.cc:3037] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.039166 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b7168ca5f4b2ba49ec83e57b48009" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:08.039335 31910 raft_consensus.cc:3037] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.041384 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b7168ca5f4b2ba49ec83e57b48009" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:08.041596 31649 raft_consensus.cc:3037] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.038707 32344 leader_election.cc:288] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 election: Requested vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:09:08.041713 31779 raft_consensus.cc:2454] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:08.041992 31910 raft_consensus.cc:2454] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:08.043318 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b7168ca5f4b2ba49ec83e57b48009" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:08.043520 31257 raft_consensus.cc:3037] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:08.044364 31649 raft_consensus.cc:2454] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:08.044888 31452 leader_election.cc:302] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:08.045337 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4b7168ca5f4b2ba49ec83e57b48009" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:08.045620 31648 raft_consensus.cc:2362] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1. Re-sending same reply.
I20240501 20:09:08.045636 31117 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70). New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.046169 31257 raft_consensus.cc:2454] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:08.046808 32344 raft_consensus.cc:2781] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:08.046959 32344 raft_consensus.cc:686] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Becoming Leader. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Running, Role: LEADER
I20240501 20:09:08.047272 32344 consensus_queue.cc:237] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } }
I20240501 20:09:08.050773 31117 catalog_manager.cc:5484] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67). New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:08.393548 32580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:08.393909 32580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:08.401082 32580 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:08.431118 32389 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 20:09:08.441394 32389 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.444191 32344 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.447674 32575 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.448186 32599 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.523236 32344 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.545782 32578 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.548218 32513 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.565363 32344 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.579717 32513 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.582994 32513 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:08.627156 32630 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 352.842 ms
time per row: 0.176421 ms
W20240501 20:09:09.374998 31715 consensus_peers.cc:485] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.71:35749: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
-----------------------------------------------------
Suppressions used:
count bytes template
28 7248 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:09.826182 31111 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39828:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:09.837257 31237 tablet_service.cc:1467] Processing CreateTablet for tablet 37fbd40e4b9a4d75994b3e138c4506d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.837740 31237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd40e4b9a4d75994b3e138c4506d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.838059 31889 tablet_service.cc:1467] Processing CreateTablet for tablet 37fbd40e4b9a4d75994b3e138c4506d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.838497 31889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd40e4b9a4d75994b3e138c4506d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.838477 31367 tablet_service.cc:1467] Processing CreateTablet for tablet 37fbd40e4b9a4d75994b3e138c4506d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.838539 31369 tablet_service.cc:1467] Processing CreateTablet for tablet 90512cd61a6647c4beb15ffcdd488da3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.838878 31367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd40e4b9a4d75994b3e138c4506d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.839097 31369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90512cd61a6647c4beb15ffcdd488da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.841192 31629 tablet_service.cc:1467] Processing CreateTablet for tablet 37fbd40e4b9a4d75994b3e138c4506d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.841588 31629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd40e4b9a4d75994b3e138c4506d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.845650 31760 tablet_service.cc:1467] Processing CreateTablet for tablet 90512cd61a6647c4beb15ffcdd488da3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.846052 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90512cd61a6647c4beb15ffcdd488da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.846007 31890 tablet_service.cc:1467] Processing CreateTablet for tablet 90512cd61a6647c4beb15ffcdd488da3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.846405 31890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90512cd61a6647c4beb15ffcdd488da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.847663 31888 tablet_service.cc:1467] Processing CreateTablet for tablet 66e635d773d34c9bbf8b2b15c4611b40 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.848037 31888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e635d773d34c9bbf8b2b15c4611b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.849115 31498 tablet_service.cc:1467] Processing CreateTablet for tablet 37fbd40e4b9a4d75994b3e138c4506d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.849498 31498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd40e4b9a4d75994b3e138c4506d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.850842 31236 tablet_service.cc:1467] Processing CreateTablet for tablet 66e635d773d34c9bbf8b2b15c4611b40 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.851230 31236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e635d773d34c9bbf8b2b15c4611b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.852408 31499 tablet_service.cc:1467] Processing CreateTablet for tablet 90512cd61a6647c4beb15ffcdd488da3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.852748 31499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90512cd61a6647c4beb15ffcdd488da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.854218 31628 tablet_service.cc:1467] Processing CreateTablet for tablet 66e635d773d34c9bbf8b2b15c4611b40 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.854594 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e635d773d34c9bbf8b2b15c4611b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.854565 31368 tablet_service.cc:1467] Processing CreateTablet for tablet 66e635d773d34c9bbf8b2b15c4611b40 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.854933 31368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e635d773d34c9bbf8b2b15c4611b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.857107 31238 tablet_service.cc:1467] Processing CreateTablet for tablet 90512cd61a6647c4beb15ffcdd488da3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.857487 31238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90512cd61a6647c4beb15ffcdd488da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.858608 31759 tablet_service.cc:1467] Processing CreateTablet for tablet 66e635d773d34c9bbf8b2b15c4611b40 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:09.858997 31759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e635d773d34c9bbf8b2b15c4611b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.873873 32676 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:09.875649 32676 tablet_bootstrap.cc:654] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.877480 32679 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:09.879371 32679 tablet_bootstrap.cc:654] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.884451 32679 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:09.884627 32679 ts_tablet_manager.cc:1367] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:09:09.885807 32676 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:09.888461 32676 ts_tablet_manager.cc:1367] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 20:09:09.889513 32676 raft_consensus.cc:348] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.890626 32679 raft_consensus.cc:348] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.891577 32679 raft_consensus.cc:374] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.891706 32676 raft_consensus.cc:374] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.891948 32676 raft_consensus.cc:729] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.892105 32679 raft_consensus.cc:729] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.892534 32676 consensus_queue.cc:260] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.893266 32680 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:09.893139 32679 consensus_queue.cc:260] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.896870 32680 tablet_bootstrap.cc:654] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.897600 32678 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:09.899276 32678 tablet_bootstrap.cc:654] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.899840 32676 ts_tablet_manager.cc:1398] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:09:09.900306 32679 ts_tablet_manager.cc:1398] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240501 20:09:09.900648 32679 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221: Bootstrap starting.
I20240501 20:09:09.902529 32676 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:09.907377 32676 tablet_bootstrap.cc:654] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.907964 32675 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:09.908353 32679 tablet_bootstrap.cc:654] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.909868 32675 tablet_bootstrap.cc:654] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.910043 32679 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221: No bootstrap required, opened a new log
I20240501 20:09:09.910202 32679 ts_tablet_manager.cc:1367] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240501 20:09:09.910928 32679 raft_consensus.cc:348] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.911177 32679 raft_consensus.cc:374] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.911247 32679 raft_consensus.cc:729] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d71e1531b4649b363236998019221, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.911502 32679 consensus_queue.cc:260] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.912153 32679 ts_tablet_manager.cc:1398] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:09.914947 32676 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:09.915092 32676 ts_tablet_manager.cc:1367] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20240501 20:09:09.915735 32676 raft_consensus.cc:348] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.915935 32676 raft_consensus.cc:374] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.916006 32676 raft_consensus.cc:729] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.916131 32678 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:09.916306 32678 ts_tablet_manager.cc:1367] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 20:09:09.916229 32676 consensus_queue.cc:260] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.916841 32676 ts_tablet_manager.cc:1398] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:09.917102 32676 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485: Bootstrap starting.
I20240501 20:09:09.917105 32678 raft_consensus.cc:348] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.918171 32677 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:09.918292 32676 tablet_bootstrap.cc:654] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.919693 32677 tablet_bootstrap.cc:654] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.919823 32678 raft_consensus.cc:374] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.919976 32675 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:09.920126 32675 ts_tablet_manager.cc:1367] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 20:09:09.920168 32676 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485: No bootstrap required, opened a new log
I20240501 20:09:09.920310 32676 ts_tablet_manager.cc:1367] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:09.919970 32678 raft_consensus.cc:729] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.920903 32675 raft_consensus.cc:348] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.921139 32675 raft_consensus.cc:374] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.920990 32678 consensus_queue.cc:260] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.921059 32676 raft_consensus.cc:348] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.921234 32675 raft_consensus.cc:729] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.921393 32676 raft_consensus.cc:374] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.921495 32676 raft_consensus.cc:729] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582c7a51c0ac43b4934078ebdf434485, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.921466 32675 consensus_queue.cc:260] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.921633 32678 ts_tablet_manager.cc:1398] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:09.921739 32676 consensus_queue.cc:260] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.921928 32678 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4: Bootstrap starting.
I20240501 20:09:09.922056 32680 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:09.922255 32680 ts_tablet_manager.cc:1367] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.004s
I20240501 20:09:09.923015 32680 raft_consensus.cc:348] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.923406 32680 raft_consensus.cc:374] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.923561 32680 raft_consensus.cc:729] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.924561 32680 consensus_queue.cc:260] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.925334 32680 ts_tablet_manager.cc:1398] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:09.925361 32675 ts_tablet_manager.cc:1398] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:09.925690 32680 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Bootstrap starting.
I20240501 20:09:09.926844 32678 tablet_bootstrap.cc:654] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.927345 32680 tablet_bootstrap.cc:654] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.928320 32676 ts_tablet_manager.cc:1398] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:09.930583 32677 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:09.930797 32677 ts_tablet_manager.cc:1367] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:09:09.931442 32678 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4: No bootstrap required, opened a new log
I20240501 20:09:09.931617 32678 ts_tablet_manager.cc:1367] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:09:09.932562 32677 raft_consensus.cc:348] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.932610 32678 raft_consensus.cc:348] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.932862 32678 raft_consensus.cc:374] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.932895 32675 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:09.932953 32678 raft_consensus.cc:729] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.933166 32677 raft_consensus.cc:374] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.933279 32677 raft_consensus.cc:729] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.933182 32678 consensus_queue.cc:260] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.933686 32678 ts_tablet_manager.cc:1398] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:09.933836 32677 consensus_queue.cc:260] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.934455 32675 tablet_bootstrap.cc:654] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.934523 32677 ts_tablet_manager.cc:1398] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:09.934808 32677 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:09.936359 32677 tablet_bootstrap.cc:654] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.939611 32677 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:09.939778 32677 ts_tablet_manager.cc:1367] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:09.940532 32677 raft_consensus.cc:348] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.940824 32677 raft_consensus.cc:374] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.940901 32677 raft_consensus.cc:729] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.941177 32677 consensus_queue.cc:260] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.941745 32677 ts_tablet_manager.cc:1398] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:09.942152 32677 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
I20240501 20:09:09.943017 32680 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: No bootstrap required, opened a new log
I20240501 20:09:09.943199 32680 ts_tablet_manager.cc:1367] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240501 20:09:09.943725 32677 tablet_bootstrap.cc:654] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.943966 32680 raft_consensus.cc:348] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.944217 32680 raft_consensus.cc:374] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.944284 32680 raft_consensus.cc:729] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9dba79a8c94a7b87bf51e9ed4b5fe1, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.944567 32680 consensus_queue.cc:260] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.945855 32680 ts_tablet_manager.cc:1398] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:09.948913 32675 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:09.949120 32675 ts_tablet_manager.cc:1367] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20240501 20:09:09.949842 32675 raft_consensus.cc:348] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.950080 32675 raft_consensus.cc:374] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.950158 32675 raft_consensus.cc:729] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.950419 32675 consensus_queue.cc:260] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.951051 32675 ts_tablet_manager.cc:1398] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:09.951390 32675 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:09.952169 32677 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986: No bootstrap required, opened a new log
I20240501 20:09:09.952316 32677 ts_tablet_manager.cc:1367] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 20:09:09.953155 32675 tablet_bootstrap.cc:654] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:09.952997 32677 raft_consensus.cc:348] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.953281 32677 raft_consensus.cc:374] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.953387 32677 raft_consensus.cc:729] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.953653 32677 consensus_queue.cc:260] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.954341 32677 ts_tablet_manager.cc:1398] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:09.956693 32653 raft_consensus.cc:482] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:09.956903 32653 raft_consensus.cc:504] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.957990 32675 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f: No bootstrap required, opened a new log
I20240501 20:09:09.957990 32653 leader_election.cc:288] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:09.958154 32675 ts_tablet_manager.cc:1367] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 20:09:09.958130 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:09.958250 31388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:09.958410 31257 raft_consensus.cc:2454] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:09.958434 31388 raft_consensus.cc:2454] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:09.959007 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:09.959193 31649 raft_consensus.cc:2454] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:09.959132 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:09.959095 32675 raft_consensus.cc:348] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.959318 32675 raft_consensus.cc:374] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:09.959353 31910 raft_consensus.cc:2454] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 0.
I20240501 20:09:09.959399 32675 raft_consensus.cc:729] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: FOLLOWER
I20240501 20:09:09.959611 31452 leader_election.cc:302] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:09.959610 32675 consensus_queue.cc:260] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:09.959898 32653 raft_consensus.cc:2781] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:09.960016 32653 raft_consensus.cc:482] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:09.960105 32653 raft_consensus.cc:3037] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.960163 32675 ts_tablet_manager.cc:1398] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:09.962692 32653 raft_consensus.cc:504] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.963629 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:09.963678 31388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:09.963845 31257 raft_consensus.cc:3037] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.963886 31388 raft_consensus.cc:3037] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.963364 32653 leader_election.cc:288] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 election: Requested vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:09.965541 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:09.965739 31649 raft_consensus.cc:3037] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.966527 31388 raft_consensus.cc:2454] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:09.966527 31257 raft_consensus.cc:2454] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:09.966967 31455 leader_election.cc:302] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, db07c3a323eb4a3c919eaf9b19b24986, ec65c12cd0724cf9aa16fcd61bbedae4; no voters:
I20240501 20:09:09.967129 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:09.967372 31910 raft_consensus.cc:3037] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:09.967388 32653 raft_consensus.cc:2781] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:09.967569 32653 raft_consensus.cc:686] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Becoming Leader. State: Replica: ec65c12cd0724cf9aa16fcd61bbedae4, State: Running, Role: LEADER
I20240501 20:09:09.967967 31649 raft_consensus.cc:2454] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:09.967900 32653 consensus_queue.cc:237] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } }
I20240501 20:09:09.969527 31910 raft_consensus.cc:2454] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec65c12cd0724cf9aa16fcd61bbedae4 in term 1.
I20240501 20:09:09.971395 31114 catalog_manager.cc:5484] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67). New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:09.995604 32461 raft_consensus.cc:482] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:09.995776 32461 raft_consensus.cc:504] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:09.996546 32461 leader_election.cc:288] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401)
I20240501 20:09:09.996907 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90512cd61a6647c4beb15ffcdd488da3" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:09.996907 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90512cd61a6647c4beb15ffcdd488da3" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:09.997121 31910 raft_consensus.cc:2454] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:09.997121 31778 raft_consensus.cc:2454] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:09.997103 31257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90512cd61a6647c4beb15ffcdd488da3" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:09.997187 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90512cd61a6647c4beb15ffcdd488da3" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" is_pre_election: true
I20240501 20:09:09.997328 31257 raft_consensus.cc:2454] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:09.997397 31518 raft_consensus.cc:2454] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:09.997689 31325 leader_election.cc:302] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:09:09.997946 32461 raft_consensus.cc:2781] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:09.998041 32461 raft_consensus.cc:482] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:09.998126 32461 raft_consensus.cc:3037] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.000043 32461 raft_consensus.cc:504] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:10.000790 32461 leader_election.cc:288] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 election: Requested vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401)
I20240501 20:09:10.001088 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90512cd61a6647c4beb15ffcdd488da3" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:10.001173 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90512cd61a6647c4beb15ffcdd488da3" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:10.001314 31910 raft_consensus.cc:3037] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.001370 31778 raft_consensus.cc:3037] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.001617 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90512cd61a6647c4beb15ffcdd488da3" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:10.001812 31258 raft_consensus.cc:3037] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.003409 31910 raft_consensus.cc:2454] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:10.004102 31778 raft_consensus.cc:2454] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:10.004371 31258 raft_consensus.cc:2454] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:10.004451 31323 leader_election.cc:302] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, d73d71e1531b4649b363236998019221, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:09:10.004622 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90512cd61a6647c4beb15ffcdd488da3" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4"
I20240501 20:09:10.004756 32461 raft_consensus.cc:2781] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:10.004871 31518 raft_consensus.cc:3037] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.005563 32461 raft_consensus.cc:686] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Becoming Leader. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Running, Role: LEADER
I20240501 20:09:10.006124 32461 consensus_queue.cc:237] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } }
I20240501 20:09:10.007524 31518 raft_consensus.cc:2454] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:10.009527 31114 catalog_manager.cc:5484] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: term changed from 0 to 1, leader changed from <none> to db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "db07c3a323eb4a3c919eaf9b19b24986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:10.033870 32461 raft_consensus.cc:482] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:10.034116 32461 raft_consensus.cc:504] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:10.035346 32461 leader_election.cc:288] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:09:10.035411 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:10.035490 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f" is_pre_election: true
I20240501 20:09:10.035655 31649 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:10.035723 31258 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:10.035387 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:10.035734 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:10.035948 31910 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:10.035975 31778 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 0.
I20240501 20:09:10.036170 31325 leader_election.cc:302] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:09:10.036463 32621 raft_consensus.cc:482] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:10.036530 32461 raft_consensus.cc:2781] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:10.036633 32461 raft_consensus.cc:482] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:10.036762 32461 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.036600 32621 raft_consensus.cc:504] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:10.037509 32621 leader_election.cc:288] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:09:10.037916 31388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986" is_pre_election: true
I20240501 20:09:10.038064 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
I20240501 20:09:10.038070 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" is_pre_election: true
I20240501 20:09:10.038256 31778 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:10.038272 31649 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:10.038465 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:09:10.038677 31191 leader_election.cc:302] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, d73d71e1531b4649b363236998019221; no voters:
I20240501 20:09:10.038724 31910 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 0.
I20240501 20:09:10.039013 32621 raft_consensus.cc:2781] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:10.039112 32621 raft_consensus.cc:482] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:10.039204 32621 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.039342 32461 raft_consensus.cc:504] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:10.039711 31388 raft_consensus.cc:2379] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 341d3bb73a5c4e8f96176a281db4a60f in current term 1: Already voted for candidate db07c3a323eb4a3c919eaf9b19b24986 in this term.
I20240501 20:09:10.040261 32461 leader_election.cc:288] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 election: Requested vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:09:10.040809 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:10.040984 31910 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.041065 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:10.041283 31649 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.042203 32621 raft_consensus.cc:504] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:10.043550 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:09:10.043576 31388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:09:10.043802 31388 raft_consensus.cc:2379] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 341d3bb73a5c4e8f96176a281db4a60f in current term 1: Already voted for candidate db07c3a323eb4a3c919eaf9b19b24986 in this term.
I20240501 20:09:10.043901 31649 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:10.044199 31910 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:10.044433 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:09:10.044569 31778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:10.044673 31909 raft_consensus.cc:2379] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 341d3bb73a5c4e8f96176a281db4a60f in current term 1: Already voted for candidate db07c3a323eb4a3c919eaf9b19b24986 in this term.
I20240501 20:09:10.044703 32621 leader_election.cc:288] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 election: Requested vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:09:10.044793 31779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:09:10.045096 31191 leader_election.cc:302] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f; no voters: 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986
I20240501 20:09:10.044800 31778 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:10.045501 31322 leader_election.cc:302] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986; no voters:
I20240501 20:09:10.045569 32621 raft_consensus.cc:2726] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:09:10.046053 31258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "db07c3a323eb4a3c919eaf9b19b24986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
I20240501 20:09:10.046283 31258 raft_consensus.cc:2379] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate db07c3a323eb4a3c919eaf9b19b24986 in current term 1: Already voted for candidate 341d3bb73a5c4e8f96176a281db4a60f in this term.
I20240501 20:09:10.046506 32461 raft_consensus.cc:2781] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:10.046669 32461 raft_consensus.cc:686] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Becoming Leader. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Running, Role: LEADER
I20240501 20:09:10.046953 32461 consensus_queue.cc:237] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } }
I20240501 20:09:10.047437 31778 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db07c3a323eb4a3c919eaf9b19b24986 in term 1.
I20240501 20:09:10.049959 31114 catalog_manager.cc:5484] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: term changed from 0 to 1, leader changed from <none> to db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "db07c3a323eb4a3c919eaf9b19b24986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:10.242996 31715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.71:35749: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240501 20:09:10.385936 32653 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:10.413834 32700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:10.414161 32700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:09:10.420943 32461 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:10.422348 32700 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:10.423427 32653 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.428403 32461 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.431450 32653 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.434340 32653 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.445470 32461 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.460120 32461 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.488059 32697 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.490100 32726 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240501 20:09:10.525144 32733 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:10.525514 32726 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:10.906857 31715 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 76991.2us
INSERT report
rows total: 2000
time total: 479.382 ms
time per row: 0.239691 ms
I20240501 20:09:11.125298 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.71:35749
--local_ip_for_outbound_sockets=127.28.136.71
--tserver_master_addrs=127.28.136.126:46583
--webserver_port=34655
--webserver_interface=127.28.136.71
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:09:11.465430 320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:11.465726 320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:11.465853 320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:11.473836 320 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:11.473968 320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:11.474143 320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.71
I20240501 20:09:11.484198 320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.71:35749
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.136.71
--webserver_port=34655
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.71
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:11.485589 320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:11.487458 320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:11.500968 325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:11.501157 326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:11.501885 328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:11.502480 320 server_base.cc:1034] running on GCE node
I20240501 20:09:11.503170 320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:11.504223 320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:11.505450 320 hybrid_clock.cc:648] HybridClock initialized: now 1714594151505397 us; error 42 us; skew 500 ppm
I20240501 20:09:11.505803 320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:11.508190 320 webserver.cc:460] Webserver started at http://127.28.136.71:34655/ using document root <none> and password file <none>
I20240501 20:09:11.509093 320 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:11.509227 320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:11.516065 320 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:09:11.519088 335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:11.520529 320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:11.520702 320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c42de85026a84ed7b4f462cf25433e01"
format_stamp: "Formatted at 2024-05-01 20:08:58 on dist-test-slave-sx0n"
I20240501 20:09:11.521579 320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:11.542312 320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:11.543236 320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:11.543567 320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:11.544838 320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:11.546922 343 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 20:09:11.552099 320 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 20:09:11.552201 320 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 20:09:11.552325 320 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 20:09:11.555738 320 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 20:09:11.555863 320 ts_tablet_manager.cc:569] Time spent register tablets: real 0.004s user 0.001s sys 0.000s
I20240501 20:09:11.556290 343 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
I20240501 20:09:11.577100 343 log.cc:824] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:11.580760 343 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:11.581431 343 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:11.582934 343 ts_tablet_manager.cc:1367] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s
I20240501 20:09:11.594861 343 raft_consensus.cc:348] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:11.596118 343 raft_consensus.cc:729] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: FOLLOWER
I20240501 20:09:11.597323 343 consensus_queue.cc:260] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } }
I20240501 20:09:11.599339 343 ts_tablet_manager.cc:1398] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20240501 20:09:11.619347 320 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.71:35749
I20240501 20:09:11.619426 452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.71:35749 every 8 connection(s)
I20240501 20:09:11.620698 320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240501 20:09:11.621977 454 tablet.cc:2369] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:11.628317 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 320
I20240501 20:09:11.628587 29217 test_util.cc:268] Using random seed: 257392412
I20240501 20:09:11.633026 453 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:09:11.633340 453 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:11.634137 453 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:09:11.637022 31113 ts_manager.cc:194] Re-registered known tserver with Master: c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749)
I20240501 20:09:11.638649 31113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.71:53445
I20240501 20:09:11.646090 29217 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 20:09:11.646580 29217 test_util.cc:268] Using random seed: 257410397
I20240501 20:09:11.662503 29217 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 20:09:11.662920 29217 test_util.cc:268] Using random seed: 257426731
I20240501 20:09:11.756155 29217 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 20:09:11.772028 29217 test_util.cc:268] Using random seed: 257535834
I20240501 20:09:11.817239 453 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
W20240501 20:09:11.894258 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 90082.5us
I20240501 20:09:11.969542 29217 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 20:09:12.044399 29217 test_util.cc:268] Using random seed: 257808189
W20240501 20:09:12.166754 31453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 82626.8us
I20240501 20:09:12.382683 29217 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240501 20:09:12.662156 569 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.28.136.126:46583, user_credentials={real_user=slave}} blocked reactor thread for 199911us
W20240501 20:09:13.370692 31192 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 44161.1us
W20240501 20:09:13.956954 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 71739.6us
W20240501 20:09:14.194593 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 49607.5us
W20240501 20:09:14.841883 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 50709.4us
W20240501 20:09:14.842705 31192 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 51421.3us
W20240501 20:09:15.020823 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 44855.1us
W20240501 20:09:15.116561 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 49329.6us
W20240501 20:09:15.568317 31843 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 188979us
W20240501 20:09:16.396903 31712 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 66320.1us
I20240501 20:09:16.445256 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:16.501063 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:16.528396 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:09:16.672837 31180 kernel_stack_watchdog.cc:198] Thread 31193 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 341ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f84683d9980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f8465047d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f84650687ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f84650643ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f84650694ff kudu::ThreadPool::Submit() at ??:0
@ 0x7f846bca3682 kudu::rpc::ReactorThread::StartConnectionNegotiation() at ??:0
@ 0x7f846bca2b8b kudu::rpc::ReactorThread::RegisterConnection() at ??:0
@ 0x7f846bcb400e kudu::rpc::RegisterConnectionTask::Run() at ??:0
@ 0x7f846bca269d kudu::rpc::ReactorThread::AsyncHandler() at ??:0
@ 0x7f846bcb4f45 ev::base<>::method_thunk<>() at ??:0
@ 0x7f8463856699 ev_invoke_pending at ??:0
@ 0x7f846bc9fba5 kudu::rpc::ReactorThread::InvokePendingCb() at ??:0
@ 0x7f84638575a1 ev_run at ??:0
W20240501 20:09:16.686070 31193 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.064s sys 0.100s
W20240501 20:09:16.686209 31193 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.511s user 0.064s sys 0.100s
W20240501 20:09:16.694180 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 147426us
I20240501 20:09:16.804914 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:16.861693 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:09:16.976409 31583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 66945.4us
I20240501 20:09:17.005162 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:09:17.726259 31583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 99061.6us
I20240501 20:09:17.735234 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:09:18.345992 31191 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 226157us
W20240501 20:09:18.412132 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 116421us
W20240501 20:09:18.844017 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 70824.9us
W20240501 20:09:19.076306 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 89161.2us
W20240501 20:09:19.478622 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 44851.2us
W20240501 20:09:20.623095 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 62488us
W20240501 20:09:20.957654 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 110794us
W20240501 20:09:21.003005 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 45411.1us
W20240501 20:09:21.175606 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 45668us
W20240501 20:09:21.484714 31712 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 135498us
W20240501 20:09:22.397159 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 71585us
I20240501 20:09:22.427773 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:09:22.523448 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 45596.2us
I20240501 20:09:22.564692 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:22.576236 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:22.736869 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:09:22.859339 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 81838.5us
I20240501 20:09:22.896392 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:23.002590 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:23.066943 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:09:23.730224 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 81888us
W20240501 20:09:24.218315 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 64466.4us
I20240501 20:09:24.383030 31907 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 1.136, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:24.386991 31513 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240501 20:09:24.387965 904 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240501 20:09:24.392503 31252 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240501 20:09:24.396384 31779 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240501 20:09:24.396899 904 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240501 20:09:24.402479 31645 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240501 20:09:24.402464 31773 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.404893 844 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240501 20:09:24.400326 904 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.410636 31645 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.412508 31115 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 reported cstate change: config changed from index -1 to 137, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.426520 31513 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:24.446717 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 108361us
I20240501 20:09:24.450742 904 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240501 20:09:24.453639 31252 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index -1 to 137, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.601816 31387 consensus_queue.cc:237] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 1.114, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:24.610271 31908 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240501 20:09:24.611078 932 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20240501 20:09:24.623492 31779 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240501 20:09:24.624146 956 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20240501 20:09:24.626606 31517 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240501 20:09:24.627838 956 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.000s
I20240501 20:09:24.628954 932 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Committing config change with OpId 1.115: config changed from index -1 to 115, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.635726 31780 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index -1 to 115, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.645107 31114 catalog_manager.cc:5484] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 reported cstate change: config changed from index -1 to 115, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "db07c3a323eb4a3c919eaf9b19b24986" committed_config { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.647433 31254 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240501 20:09:24.649521 31909 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index -1 to 115, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:24.658387 31846 consensus_peers.cc:485] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7faad10e555240378c4deeb23188128a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:24.685660 956 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.000s
I20240501 20:09:24.689909 31255 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index -1 to 115, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:24.696990 31517 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index -1 to 115, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:24.861810 31325 consensus_peers.cc:485] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90512cd61a6647c4beb15ffcdd488da3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:24.996181 994 ts_tablet_manager.cc:902] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:09:25.159811 1004 ts_tablet_manager.cc:902] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527)
I20240501 20:09:25.208902 31905 consensus_queue.cc:237] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } }
I20240501 20:09:25.209416 994 tablet_copy_client.cc:308] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:37941
I20240501 20:09:25.212725 31776 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240501 20:09:25.213913 821 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240501 20:09:25.220453 31385 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240501 20:09:25.262100 31648 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240501 20:09:25.266196 989 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 85, Time since last communication: 0.000s
W20240501 20:09:25.276978 31846 consensus_peers.cc:485] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 -> Peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Couldn't send request to peer 341d3bb73a5c4e8f96176a281db4a60f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 791393d238c3413d9472eecffbf1438a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:25.279796 904 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240501 20:09:25.280854 821 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } } }
I20240501 20:09:25.289325 31648 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } } }
I20240501 20:09:25.291138 31110 catalog_manager.cc:5484] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index -1 to 87, NON_VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) added. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:25.333062 31777 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } } }
I20240501 20:09:25.385928 31385 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } } }
I20240501 20:09:25.460629 31930 tablet_copy_service.cc:140] P 582c7a51c0ac43b4934078ebdf434485: Received BeginTabletCopySession request for tablet 7faad10e555240378c4deeb23188128a from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:34147)
I20240501 20:09:25.460945 31930 tablet_copy_service.cc:161] P 582c7a51c0ac43b4934078ebdf434485: Beginning new tablet copy session on tablet 7faad10e555240378c4deeb23188128a from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:34147: session id = c42de85026a84ed7b4f462cf25433e01-7faad10e555240378c4deeb23188128a
I20240501 20:09:25.465929 31930 tablet_copy_source_session.cc:205] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:25.470551 994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faad10e555240378c4deeb23188128a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:09:25.489943 31453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 309273us
I20240501 20:09:25.491353 31514 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240501 20:09:25.492592 904 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 85, Time since last communication: 0.000s
I20240501 20:09:25.497579 31514 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } } }
I20240501 20:09:25.502429 994 tablet_copy_client.cc:783] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:25.502902 994 tablet_copy_client.cc:647] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:25.507592 994 tablet_copy_client.cc:523] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 20:09:25.508702 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 62262.6us
I20240501 20:09:25.519353 994 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
W20240501 20:09:25.653192 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 71382.8us
I20240501 20:09:25.925242 1022 ts_tablet_manager.cc:902] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: Initiating tablet copy from peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:09:25.949012 31649 consensus_queue.cc:237] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:25.952976 31909 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240501 20:09:25.953351 31776 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240501 20:09:25.953998 936 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
W20240501 20:09:25.954749 1004 thread.cc:636] tablet-download-pool-90512cd61a6647c4beb15ffcdd488da3 [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.134s sys 0.227s
I20240501 20:09:25.955673 936 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
W20240501 20:09:25.955962 1004 thread.cc:602] tablet-download-pool-90512cd61a6647c4beb15ffcdd488da3 [worker] (thread pool) Time spent starting thread: real 0.657s user 0.134s sys 0.227s
I20240501 20:09:25.956164 1004 tablet_copy_client.cc:308] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:42527
I20240501 20:09:25.961827 955 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:25.963738 31909 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:25.976491 31115 catalog_manager.cc:5484] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index -1 to 81, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:25.984889 31514 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240501 20:09:25.993880 936 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 20:09:25.997606 31776 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:26.069669 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 65768.5us
I20240501 20:09:26.073671 31385 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240501 20:09:26.080262 936 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 20:09:26.092399 31514 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.094544 31409 tablet_copy_service.cc:140] P db07c3a323eb4a3c919eaf9b19b24986: Received BeginTabletCopySession request for tablet 90512cd61a6647c4beb15ffcdd488da3 from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:36613)
I20240501 20:09:26.094828 31409 tablet_copy_service.cc:161] P db07c3a323eb4a3c919eaf9b19b24986: Beginning new tablet copy session on tablet 90512cd61a6647c4beb15ffcdd488da3 from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:36613: session id = c42de85026a84ed7b4f462cf25433e01-90512cd61a6647c4beb15ffcdd488da3
I20240501 20:09:26.098629 31409 tablet_copy_source_session.cc:205] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:26.104202 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90512cd61a6647c4beb15ffcdd488da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.111490 1004 tablet_copy_client.cc:783] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:26.111847 1004 tablet_copy_client.cc:647] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:26.121488 1004 tablet_copy_client.cc:523] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:26.122244 31385 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index -1 to 81, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.127887 1004 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
I20240501 20:09:26.175585 31513 consensus_queue.cc:237] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 1.162, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:26.180039 31773 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 162. Preceding OpId from leader: term: 1 index: 163. (index mismatch)
I20240501 20:09:26.181119 735 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 163, Last known committed idx: 161, Time since last communication: 0.000s
I20240501 20:09:26.182220 31649 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 161. Preceding OpId from leader: term: 1 index: 163. (index mismatch)
I20240501 20:09:26.183219 31909 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 162. Preceding OpId from leader: term: 1 index: 163. (index mismatch)
I20240501 20:09:26.186182 946 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 163, Last known committed idx: 161, Time since last communication: 0.000s
I20240501 20:09:26.192844 31255 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 161. Preceding OpId from leader: term: 1 index: 163. (index mismatch)
I20240501 20:09:26.193456 1006 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 163, Last known committed idx: 161, Time since last communication: 0.000s
I20240501 20:09:26.195602 1022 tablet_copy_client.cc:308] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:37941
I20240501 20:09:26.196487 31930 tablet_copy_service.cc:140] P 582c7a51c0ac43b4934078ebdf434485: Received BeginTabletCopySession request for tablet 791393d238c3413d9472eecffbf1438a from peer 341d3bb73a5c4e8f96176a281db4a60f ({username='slave'} at 127.28.136.65:40039)
I20240501 20:09:26.196643 31930 tablet_copy_service.cc:161] P 582c7a51c0ac43b4934078ebdf434485: Beginning new tablet copy session on tablet 791393d238c3413d9472eecffbf1438a from peer 341d3bb73a5c4e8f96176a281db4a60f at {username='slave'} at 127.28.136.65:40039: session id = 341d3bb73a5c4e8f96176a281db4a60f-791393d238c3413d9472eecffbf1438a
I20240501 20:09:26.199558 31930 tablet_copy_source_session.cc:205] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:26.209594 734 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.163: config changed from index -1 to 163, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.211067 31648 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.163: config changed from index -1 to 163, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.207597 1022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 791393d238c3413d9472eecffbf1438a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.234874 31779 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.163: config changed from index -1 to 163, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.253196 31115 catalog_manager.cc:5484] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 reported cstate change: config changed from index -1 to 163, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.269609 1006 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 163, Last known committed idx: 160, Time since last communication: 0.000s
I20240501 20:09:26.272423 31910 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.163: config changed from index -1 to 163, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.281129 31252 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.163: config changed from index -1 to 163, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:26.306075 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 53062.4us
W20240501 20:09:26.366253 31455 consensus_peers.cc:485] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef4b7168ca5f4b2ba49ec83e57b48009. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:09:26.377143 31585 consensus_peers.cc:485] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e93ddb68383c43089544a6f0549c83e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:26.397534 994 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=149 overwritten=0 applied=149 ignored=0} inserts{seen=834 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:26.398126 994 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:26.398584 994 ts_tablet_manager.cc:1367] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 0.879s user 0.193s sys 0.029s
I20240501 20:09:26.400257 994 raft_consensus.cc:348] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:26.400688 994 raft_consensus.cc:729] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:26.400996 994 consensus_queue.cc:260] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 149, Last appended: 1.149, Last appended by leader: 149, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:26.410125 1022 tablet_copy_client.cc:783] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:26.410784 1022 tablet_copy_client.cc:647] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:26.429436 994 ts_tablet_manager.cc:1398] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.030s user 0.007s sys 0.001s
I20240501 20:09:26.431605 31930 tablet_copy_service.cc:342] P 582c7a51c0ac43b4934078ebdf434485: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-7faad10e555240378c4deeb23188128a received from {username='slave'} at 127.28.136.71:34147
I20240501 20:09:26.431804 31930 tablet_copy_service.cc:433] P 582c7a51c0ac43b4934078ebdf434485: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-7faad10e555240378c4deeb23188128a on tablet 7faad10e555240378c4deeb23188128a with peer c42de85026a84ed7b4f462cf25433e01
W20240501 20:09:26.449280 31583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 77697.8us
W20240501 20:09:26.456274 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 73833.9us
W20240501 20:09:26.456982 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 90907.2us
I20240501 20:09:26.458050 1022 tablet_copy_client.cc:523] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 20:09:26.558894 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 101413us
W20240501 20:09:26.563548 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 91590.5us
I20240501 20:09:26.614660 407 raft_consensus.cc:1203] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.136->[1.137-1.153] Dedup: 1.149->[1.150-1.153]
W20240501 20:09:26.647887 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 70866.5us
I20240501 20:09:26.652300 1022 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap starting.
I20240501 20:09:26.693795 994 ts_tablet_manager.cc:902] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:26.726011 994 tablet_copy_client.cc:308] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:42361
W20240501 20:09:26.771188 1004 log.cc:924] Time spent T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240501 20:09:26.783178 31539 tablet_copy_service.cc:140] P ec65c12cd0724cf9aa16fcd61bbedae4: Received BeginTabletCopySession request for tablet ef4b7168ca5f4b2ba49ec83e57b48009 from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:45421)
I20240501 20:09:26.783499 31539 tablet_copy_service.cc:161] P ec65c12cd0724cf9aa16fcd61bbedae4: Beginning new tablet copy session on tablet ef4b7168ca5f4b2ba49ec83e57b48009 from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:45421: session id = c42de85026a84ed7b4f462cf25433e01-ef4b7168ca5f4b2ba49ec83e57b48009
I20240501 20:09:26.787194 31539 tablet_copy_source_session.cc:205] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:26.801157 994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4b7168ca5f4b2ba49ec83e57b48009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:26.803386 821 raft_consensus.cc:1050] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:26.804420 821 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 1.157, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:26.814425 31254 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240501 20:09:26.814564 406 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240501 20:09:26.817253 844 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240501 20:09:26.817519 957 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240501 20:09:26.816598 31255 consensus_queue.cc:237] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:26.824606 994 tablet_copy_client.cc:783] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:26.824894 994 tablet_copy_client.cc:647] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:26.828158 31514 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240501 20:09:26.829736 31777 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240501 20:09:26.829881 31387 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240501 20:09:26.830361 1068 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240501 20:09:26.830876 789 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240501 20:09:26.834208 1021 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.834859 31514 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.839272 994 tablet_copy_client.cc:523] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:26.846441 31777 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.851907 994 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
I20240501 20:09:26.851636 31113 catalog_manager.cc:5484] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index -1 to 103, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:26.855906 31514 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240501 20:09:26.856552 989 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240501 20:09:26.859850 844 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.158: config changed from index 137 to 158, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:26.867156 31513 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 137 to 158, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:26.880038 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 67649.1us
I20240501 20:09:26.890348 31646 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240501 20:09:26.911562 31777 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240501 20:09:26.912266 989 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240501 20:09:26.914585 31115 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 137 to 158, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:26.915135 31775 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 137 to 158, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:26.917121 31253 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 137 to 158, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:26.920804 1067 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.004s
I20240501 20:09:26.921713 406 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 137 to 158, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:26.931394 31385 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:26.942704 821 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 153, Time since last communication: 0.000s
I20240501 20:09:26.958690 31646 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240501 20:09:26.961149 1094 ts_tablet_manager.cc:902] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:27.016280 31646 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index 137 to 158, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.019490 1094 tablet_copy_client.cc:308] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:44049
I20240501 20:09:27.030958 1068 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240501 20:09:27.056010 31647 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index -1 to 103, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:27.074216 31194 consensus_peers.cc:485] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3b0a8894c0f46c683461a1857ff27ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:27.106556 31906 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 162, Committed index: 162, Last appended: 1.163, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:27.115603 406 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 163. Preceding OpId from leader: term: 1 index: 164. (index mismatch)
I20240501 20:09:27.116814 989 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 162, Time since last communication: 0.000s
I20240501 20:09:27.126363 31519 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 163. Preceding OpId from leader: term: 1 index: 164. (index mismatch)
I20240501 20:09:27.127878 821 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 162, Time since last communication: 0.000s
I20240501 20:09:27.131471 989 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.164: config changed from index 158 to 164, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.137876 407 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 158 to 164, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.138731 31779 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 162. Preceding OpId from leader: term: 1 index: 164. (index mismatch)
I20240501 20:09:27.141093 989 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 162, Time since last communication: 0.000s
I20240501 20:09:27.149396 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7faad10e555240378c4deeb23188128a with cas_config_opid_index 158: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:27.150458 31514 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 158 to 164, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.150843 31115 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 158 to 164, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:27.151322 31777 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 158 to 164, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.158442 31237 tablet_service.cc:1514] Processing DeleteTablet for tablet 7faad10e555240378c4deeb23188128a with delete_type TABLET_DATA_TOMBSTONED (TS 341d3bb73a5c4e8f96176a281db4a60f not found in new config with opid_index 164) from {username='slave'} at 127.0.0.1:51884
I20240501 20:09:27.190237 1119 tablet_replica.cc:330] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: stopping tablet replica
I20240501 20:09:27.198207 31647 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 162. Preceding OpId from leader: term: 1 index: 164. (index mismatch)
I20240501 20:09:27.193545 1022 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=93 ignored=0} inserts{seen=724 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:27.199182 1022 tablet_bootstrap.cc:492] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: Bootstrap complete.
I20240501 20:09:27.199595 1022 ts_tablet_manager.cc:1367] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: Time spent bootstrapping tablet: real 0.547s user 0.169s sys 0.056s
I20240501 20:09:27.201030 1022 raft_consensus.cc:348] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } }
I20240501 20:09:27.201370 1022 raft_consensus.cc:729] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Initialized, Role: LEARNER
I20240501 20:09:27.201261 989 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 162, Time since last communication: 0.000s
I20240501 20:09:27.201607 1022 consensus_queue.cc:260] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 1.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: NON_VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: true } }
I20240501 20:09:27.202291 1022 ts_tablet_manager.cc:1398] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:27.204036 31930 tablet_copy_service.cc:342] P 582c7a51c0ac43b4934078ebdf434485: Request end of tablet copy session 341d3bb73a5c4e8f96176a281db4a60f-791393d238c3413d9472eecffbf1438a received from {username='slave'} at 127.28.136.65:40039
I20240501 20:09:27.204155 31930 tablet_copy_service.cc:433] P 582c7a51c0ac43b4934078ebdf434485: ending tablet copy session 341d3bb73a5c4e8f96176a281db4a60f-791393d238c3413d9472eecffbf1438a on tablet 791393d238c3413d9472eecffbf1438a with peer 341d3bb73a5c4e8f96176a281db4a60f
I20240501 20:09:27.217675 1119 raft_consensus.cc:2229] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:27.227326 31773 raft_consensus.cc:1203] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.162->[1.163-1.163] Dedup: 1.163->[]
I20240501 20:09:27.233420 1004 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=129 overwritten=0 applied=129 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:27.233960 1004 tablet_bootstrap.cc:492] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:27.234259 1004 ts_tablet_manager.cc:1367] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 1.106s user 0.176s sys 0.021s
I20240501 20:09:27.235035 1004 raft_consensus.cc:348] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:27.235299 1004 raft_consensus.cc:729] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:27.235543 1004 consensus_queue.cc:260] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 129, Last appended: 1.129, Last appended by leader: 129, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
W20240501 20:09:27.252214 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 51962.9us
I20240501 20:09:27.256558 1004 ts_tablet_manager.cc:1398] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240501 20:09:27.260859 31409 tablet_copy_service.cc:342] P db07c3a323eb4a3c919eaf9b19b24986: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-90512cd61a6647c4beb15ffcdd488da3 received from {username='slave'} at 127.28.136.71:36613
I20240501 20:09:27.261068 31409 tablet_copy_service.cc:433] P db07c3a323eb4a3c919eaf9b19b24986: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-90512cd61a6647c4beb15ffcdd488da3 on tablet 90512cd61a6647c4beb15ffcdd488da3 with peer c42de85026a84ed7b4f462cf25433e01
I20240501 20:09:27.288268 31647 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 158 to 164, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.295646 31669 tablet_copy_service.cc:140] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Received BeginTabletCopySession request for tablet e93ddb68383c43089544a6f0549c83e2 from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:49171)
I20240501 20:09:27.295948 31669 tablet_copy_service.cc:161] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Beginning new tablet copy session on tablet e93ddb68383c43089544a6f0549c83e2 from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:49171: session id = c42de85026a84ed7b4f462cf25433e01-e93ddb68383c43089544a6f0549c83e2
I20240501 20:09:27.299460 31669 tablet_copy_source_session.cc:205] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:27.301782 1119 pending_rounds.cc:62] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: Trying to abort 1 pending ops.
W20240501 20:09:27.302939 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 62834.4us
I20240501 20:09:27.304296 1094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93ddb68383c43089544a6f0549c83e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:27.304862 1119 pending_rounds.cc:68] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: Aborting op as it isn't in flight: id { term: 1 index: 163 } timestamp: 7022977708441444352 op_type: WRITE_OP write_request { tablet_id: "7faad10e555240378c4deeb23188128a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\2220b\034\277\22485\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""5d50b9c0bda63839" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977708262899712 authz_token { token_data: "\010\224\305\312\261\006\"3\n\005slave\022*\n f20b4eff405e4a3d85fddbfd4f9a2474\020\001\030\001 \001(\001" signature: "<redacted>""z\221^\352\021%sP;n\270\321\315\033\363CG\252\321\355\330\360\341\214k\340\204\022|\014\037\354\014\246V\230\265\300l\212\320\016\323\247r\346\\&D\360\334#\037\213`a\375\363\177\206ZIn\327" signing_key_seq_num: 0 } } request_id { client_id: "c0c6bc589a72478fa044ca67d9113a74" seq_no: 472 first_incomplete_seq_no: 472 attempt_no: 0 }
I20240501 20:09:27.311955 1094 tablet_copy_client.cc:783] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:27.312247 1094 tablet_copy_client.cc:647] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:27.322628 1119 raft_consensus.cc:2258] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:27.327490 1119 ts_tablet_manager.cc:1875] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:27.417418 31258 raft_consensus.cc:1203] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.86->[1.87-1.95] Dedup: 1.93->[1.94-1.95]
I20240501 20:09:27.423851 1004 ts_tablet_manager.cc:902] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401)
I20240501 20:09:27.438405 407 raft_consensus.cc:1203] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.114->[1.115-1.136] Dedup: 1.129->[1.130-1.136]
I20240501 20:09:27.479318 1094 tablet_copy_client.cc:523] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:27.484969 956 raft_consensus.cc:1050] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:27.486053 956 consensus_queue.cc:237] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 1.137, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:27.489917 31773 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:27.490952 1115 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:27.491122 406 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:27.491961 31255 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:27.492841 584 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:27.493346 584 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:27.500335 1115 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Committing config change with OpId 1.138: config changed from index 115 to 138, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.505276 31255 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index 115 to 138, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.514194 1094 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
I20240501 20:09:27.529414 31773 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index 115 to 138, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.530651 406 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index 115 to 138, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.535562 31108 catalog_manager.cc:5484] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: config changed from index 115 to 138, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db07c3a323eb4a3c919eaf9b19b24986" committed_config { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:27.590934 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 49735.2us
I20240501 20:09:27.591866 31519 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
W20240501 20:09:27.601101 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 51538.8us
I20240501 20:09:27.602588 31910 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:27.604249 1115 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 136, Time since last communication: 0.000s
I20240501 20:09:27.605167 1115 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 136, Time since last communication: 0.000s
I20240501 20:09:27.607064 31910 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index 115 to 138, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.607520 1119 ts_tablet_manager.cc:1888] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.163
I20240501 20:09:27.607836 1119 log.cc:1196] T 7faad10e555240378c4deeb23188128a P 341d3bb73a5c4e8f96176a281db4a60f: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/7faad10e555240378c4deeb23188128a
I20240501 20:09:27.610193 31104 catalog_manager.cc:4830] TS 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): tablet 7faad10e555240378c4deeb23188128a (table rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]) successfully deleted
I20240501 20:09:27.651381 994 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=167 overwritten=0 applied=167 ignored=0} inserts{seen=798 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:27.651948 994 tablet_bootstrap.cc:492] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:27.652262 994 ts_tablet_manager.cc:1367] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 0.800s user 0.222s sys 0.058s
I20240501 20:09:27.653208 994 raft_consensus.cc:348] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:27.653553 994 raft_consensus.cc:729] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:27.653824 994 consensus_queue.cc:260] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 167, Last appended: 1.167, Last appended by leader: 167, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 163 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:27.655982 994 ts_tablet_manager.cc:1398] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 20:09:27.662039 31514 raft_consensus.cc:1203] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.136->[1.137-1.138] Dedup: 1.137->[1.138-1.138]
I20240501 20:09:27.664288 31539 tablet_copy_service.cc:342] P ec65c12cd0724cf9aa16fcd61bbedae4: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-ef4b7168ca5f4b2ba49ec83e57b48009 received from {username='slave'} at 127.28.136.71:45421
I20240501 20:09:27.664471 31539 tablet_copy_service.cc:433] P ec65c12cd0724cf9aa16fcd61bbedae4: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-ef4b7168ca5f4b2ba49ec83e57b48009 on tablet ef4b7168ca5f4b2ba49ec83e57b48009 with peer c42de85026a84ed7b4f462cf25433e01
I20240501 20:09:27.664592 31514 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index 115 to 138, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:27.734026 844 raft_consensus.cc:1050] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485: attempting to promote NON_VOTER 341d3bb73a5c4e8f96176a281db4a60f to VOTER
I20240501 20:09:27.735034 844 consensus_queue.cc:237] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } }
I20240501 20:09:27.738598 31258 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEARNER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 20:09:27.739101 821 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 20:09:27.743191 31519 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 20:09:27.744002 31776 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 20:09:27.743939 821 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 20:09:27.744727 821 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 20:09:27.756636 31386 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 20:09:27.759585 989 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 96, Time since last communication: 0.000s
I20240501 20:09:27.764966 844 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 87 to 99, 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.773355 31115 catalog_manager.cc:5484] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 87 to 99, 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:27.771493 31517 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 87 to 99, 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.769603 31776 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 87 to 99, 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.801493 31388 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 87 to 99, 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.810034 31647 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 20:09:27.820194 821 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 96, Time since last communication: 0.000s
I20240501 20:09:27.830574 31646 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 87 to 99, 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.884934 31906 consensus_queue.cc:237] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } }
I20240501 20:09:27.893431 31773 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240501 20:09:27.894004 989 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240501 20:09:27.894472 31517 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240501 20:09:27.899591 821 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240501 20:09:27.900215 31255 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 87 to 99, 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.921981 989 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 99 to 102, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.927623 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 791393d238c3413d9472eecffbf1438a with cas_config_opid_index 99: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:27.929113 31776 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 99 to 102, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.930179 31108 catalog_manager.cc:5484] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 99 to 102, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:27.931892 31517 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 99 to 102, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:27.934715 31367 tablet_service.cc:1514] Processing DeleteTablet for tablet 791393d238c3413d9472eecffbf1438a with delete_type TABLET_DATA_TOMBSTONED (TS db07c3a323eb4a3c919eaf9b19b24986 not found in new config with opid_index 102) from {username='slave'} at 127.0.0.1:51676
I20240501 20:09:27.946878 31646 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240501 20:09:27.981251 844 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240501 20:09:27.992417 31253 raft_consensus.cc:1261] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240501 20:09:28.001686 844 consensus_queue.cc:1035] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240501 20:09:28.003418 31253 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 99 to 102, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:28.053607 31385 consensus_queue.cc:237] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 138, Committed index: 138, Last appended: 1.138, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:28.064792 406 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240501 20:09:28.065407 1115 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240501 20:09:28.066206 31779 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240501 20:09:28.066812 1115 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240501 20:09:28.071624 1115 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Committing config change with OpId 1.139: config changed from index 138 to 139, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.078622 31104 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 90512cd61a6647c4beb15ffcdd488da3 with cas_config_opid_index 138: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:28.078722 406 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 138 to 139, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.084175 31113 catalog_manager.cc:5484] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 reported cstate change: config changed from index 138 to 139, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New cstate: current_term: 1 leader_uuid: "db07c3a323eb4a3c919eaf9b19b24986" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.088862 31888 tablet_service.cc:1514] Processing DeleteTablet for tablet 90512cd61a6647c4beb15ffcdd488da3 with delete_type TABLET_DATA_TOMBSTONED (TS 582c7a51c0ac43b4934078ebdf434485 not found in new config with opid_index 139) from {username='slave'} at 127.0.0.1:46678
I20240501 20:09:28.092957 1152 tablet_replica.cc:330] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: stopping tablet replica
I20240501 20:09:28.093531 1152 raft_consensus.cc:2229] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:28.094406 1152 raft_consensus.cc:2258] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:28.094871 31777 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 138 to 139, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.098012 1152 ts_tablet_manager.cc:1875] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:09:28.100284 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 54314.2us
I20240501 20:09:28.105245 1004 tablet_copy_client.cc:308] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:33401
I20240501 20:09:28.116724 1152 ts_tablet_manager.cc:1888] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.138
I20240501 20:09:28.120290 1152 log.cc:1196] T 90512cd61a6647c4beb15ffcdd488da3 P 582c7a51c0ac43b4934078ebdf434485: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/wals/90512cd61a6647c4beb15ffcdd488da3
I20240501 20:09:28.122720 31101 catalog_manager.cc:4830] TS 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): tablet 90512cd61a6647c4beb15ffcdd488da3 (table rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]) successfully deleted
I20240501 20:09:28.128974 406 raft_consensus.cc:1203] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.162->[1.163-1.177] Dedup: 1.167->[1.168-1.177]
I20240501 20:09:28.133724 31514 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240501 20:09:28.145578 31253 raft_consensus.cc:1261] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240501 20:09:28.153416 1114 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240501 20:09:28.173499 1115 consensus_queue.cc:1035] T 90512cd61a6647c4beb15ffcdd488da3 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 138, Time since last communication: 0.000s
I20240501 20:09:28.194128 31644 raft_consensus.cc:2932] T 791393d238c3413d9472eecffbf1438a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 99 to 102, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { promote: false } } }
I20240501 20:09:28.237334 31518 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 138 to 139, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.258185 1146 tablet_replica.cc:330] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: stopping tablet replica
I20240501 20:09:28.308178 31258 raft_consensus.cc:2932] T 90512cd61a6647c4beb15ffcdd488da3 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index 138 to 139, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:28.344849 972 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.041s sys 0.074s
W20240501 20:09:28.344985 972 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.622s user 0.041s sys 0.074s
W20240501 20:09:28.346457 1094 log.cc:924] Time spent T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20240501 20:09:28.357417 1094 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=87 ignored=0} inserts{seen=772 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:09:28.357975 1094 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:28.358230 1094 ts_tablet_manager.cc:1367] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 0.844s user 0.153s sys 0.003s
I20240501 20:09:28.358952 1094 raft_consensus.cc:348] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:28.359598 1094 raft_consensus.cc:729] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:28.359840 1094 consensus_queue.cc:260] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 87, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:28.360469 1094 ts_tablet_manager.cc:1398] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:28.405390 31669 tablet_copy_service.cc:342] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-e93ddb68383c43089544a6f0549c83e2 received from {username='slave'} at 127.28.136.71:49171
I20240501 20:09:28.420567 31669 tablet_copy_service.cc:433] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-e93ddb68383c43089544a6f0549c83e2 on tablet e93ddb68383c43089544a6f0549c83e2 with peer c42de85026a84ed7b4f462cf25433e01
I20240501 20:09:28.425132 1146 raft_consensus.cc:2229] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:28.465559 1146 raft_consensus.cc:2258] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:28.503275 1146 ts_tablet_manager.cc:1875] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:28.515900 1146 ts_tablet_manager.cc:1888] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.101
I20240501 20:09:28.516181 1146 log.cc:1196] T 791393d238c3413d9472eecffbf1438a P db07c3a323eb4a3c919eaf9b19b24986: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/791393d238c3413d9472eecffbf1438a
I20240501 20:09:28.549829 31278 tablet_copy_service.cc:140] P 341d3bb73a5c4e8f96176a281db4a60f: Received BeginTabletCopySession request for tablet a3b0a8894c0f46c683461a1857ff27ff from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:47593)
I20240501 20:09:28.550215 31278 tablet_copy_service.cc:161] P 341d3bb73a5c4e8f96176a281db4a60f: Beginning new tablet copy session on tablet a3b0a8894c0f46c683461a1857ff27ff from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:47593: session id = c42de85026a84ed7b4f462cf25433e01-a3b0a8894c0f46c683461a1857ff27ff
I20240501 20:09:28.559471 31278 tablet_copy_source_session.cc:205] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:28.565227 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0a8894c0f46c683461a1857ff27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.571674 1004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.28.136.65:46831 (request call id 1) took 1148 ms. Trace:
I20240501 20:09:28.577935 31104 catalog_manager.cc:4830] TS db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): tablet 791393d238c3413d9472eecffbf1438a (table rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]) successfully deleted
W20240501 20:09:28.578187 31104 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 60849us
I20240501 20:09:28.571854 1004 rpcz_store.cc:276] 0501 20:09:27.423450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:09:27.423520 (+ 70us) service_pool.cc:225] Handling call
0501 20:09:27.423947 (+ 427us) ts_tablet_manager.cc:903] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401)
0501 20:09:28.564680 (+1140733us) tablet_copy_client.cc:353] Tablet copy session begun
0501 20:09:28.568330 (+ 3650us) tablet_metadata.cc:641] Metadata flushed
0501 20:09:28.568438 (+ 108us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 20:09:28.571389 (+ 2951us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 20:09:28.571503 (+ 114us) tablet_replica.cc:205] Creating consensus instance
0501 20:09:28.571602 (+ 99us) ts_tablet_manager.cc:958] Replica registered.
0501 20:09:28.571627 (+ 25us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":86,"tablet-copy.queue_time_us":60,"thread_start_us":680884,"threads_started":4}
I20240501 20:09:28.589103 1004 tablet_copy_client.cc:783] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:28.589681 1004 tablet_copy_client.cc:647] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:28.594249 1004 tablet_copy_client.cc:523] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:28.599248 1004 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
I20240501 20:09:28.608621 1122 raft_consensus.cc:1050] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:28.609920 1122 consensus_queue.cc:237] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 1.181, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:28.618489 31258 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240501 20:09:28.618958 1053 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240501 20:09:28.627429 31777 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240501 20:09:28.629314 1053 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240501 20:09:28.652607 31909 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240501 20:09:28.659140 1006 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240501 20:09:28.616993 406 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240501 20:09:28.668819 406 raft_consensus.cc:1203] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.80->[1.81-1.94] Dedup: 1.88->[1.89-1.94]
I20240501 20:09:28.671592 1006 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
W20240501 20:09:28.672784 31609 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.010s sys 0.038s
W20240501 20:09:28.672958 31609 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.622s user 0.010s sys 0.038s
I20240501 20:09:28.672968 735 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.182: config changed from index 163 to 182, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.677708 407 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 163 to 182, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.684206 31909 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 163 to 182, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.700768 31646 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240501 20:09:28.702491 31108 catalog_manager.cc:5484] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 163 to 182, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:28.724692 1060 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.020s sys 0.028s
I20240501 20:09:28.734577 1006 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 179, Time since last communication: 0.000s
W20240501 20:09:28.761427 1060 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.711s user 0.020s sys 0.029s
I20240501 20:09:28.768877 31775 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 163 to 182, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.820531 31647 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 163 to 182, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.823173 31517 consensus_queue.cc:237] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 182, Committed index: 182, Last appended: 1.182, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:28.900182 31773 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240501 20:09:28.900790 1174 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240501 20:09:28.936357 406 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240501 20:09:28.941933 31908 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240501 20:09:28.950753 1186 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240501 20:09:28.953820 31258 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 163 to 182, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.956267 1186 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240501 20:09:28.972995 955 raft_consensus.cc:1050] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:28.974117 955 consensus_queue.cc:237] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:28.978734 1186 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:28.981772 31254 raft_consensus.cc:1261] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240501 20:09:28.983856 31385 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 20:09:28.984999 407 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 20:09:28.989331 1064 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 20:09:28.991196 31514 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 94. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 20:09:28.992242 1122 consensus_queue.cc:1035] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240501 20:09:28.993011 1064 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 20:09:28.995695 1064 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 20:09:28.999838 31775 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 20:09:29.000134 31908 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.005400 1064 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 20:09:29.007120 31254 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.008229 1058 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index 81 to 96, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.012753 31115 catalog_manager.cc:5484] T ef4b7168ca5f4b2ba49ec83e57b48009 P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index 182 to 183, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.014496 31513 raft_consensus.cc:1203] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.94->[1.95-1.96] Dedup: 1.95->[1.96-1.96]
I20240501 20:09:29.014701 31775 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 81 to 96, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.017027 31113 catalog_manager.cc:5484] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 81 to 96, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:29.018087 406 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.022444 31385 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 81 to 96, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.023474 31102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef4b7168ca5f4b2ba49ec83e57b48009 with cas_config_opid_index 182: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:29.030655 31514 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 81 to 96, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.030238 407 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 81 to 96, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.040071 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet ef4b7168ca5f4b2ba49ec83e57b48009 with delete_type TABLET_DATA_TOMBSTONED (TS ce9dba79a8c94a7b87bf51e9ed4b5fe1 not found in new config with opid_index 183) from {username='slave'} at 127.0.0.1:44544
I20240501 20:09:29.052688 31775 raft_consensus.cc:2932] T ef4b7168ca5f4b2ba49ec83e57b48009 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.089716 1189 tablet_replica.cc:330] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: stopping tablet replica
I20240501 20:09:29.092614 1189 raft_consensus.cc:2229] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:29.094913 1189 raft_consensus.cc:2258] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:29.137027 1189 ts_tablet_manager.cc:1875] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:09:29.237708 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 47760.8us
I20240501 20:09:29.274057 1189 ts_tablet_manager.cc:1888] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.182
I20240501 20:09:29.277740 1189 log.cc:1196] T ef4b7168ca5f4b2ba49ec83e57b48009 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/ef4b7168ca5f4b2ba49ec83e57b48009
I20240501 20:09:29.373991 31101 catalog_manager.cc:4830] TS ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): tablet ef4b7168ca5f4b2ba49ec83e57b48009 (table rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]) successfully deleted
W20240501 20:09:29.497282 31843 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.006s sys 0.005s
W20240501 20:09:29.497443 31843 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.576s user 0.006s sys 0.005s
I20240501 20:09:29.547386 31910 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 20:09:29.548326 955 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 94, Time since last communication: 0.000s
I20240501 20:09:29.549921 31910 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index 81 to 96, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.709988 1004 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=112 overwritten=0 applied=111 ignored=0} inserts{seen=783 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:09:29.717360 1004 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:29.718364 1004 ts_tablet_manager.cc:1367] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 1.119s user 0.167s sys 0.008s
I20240501 20:09:29.719239 1004 raft_consensus.cc:348] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:29.725414 1004 raft_consensus.cc:729] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:29.726939 1004 consensus_queue.cc:260] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 111, Last appended: 1.112, Last appended by leader: 112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:29.785483 1004 ts_tablet_manager.cc:1398] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.067s user 0.015s sys 0.001s
I20240501 20:09:29.786473 31278 tablet_copy_service.cc:342] P 341d3bb73a5c4e8f96176a281db4a60f: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-a3b0a8894c0f46c683461a1857ff27ff received from {username='slave'} at 127.28.136.71:47593
I20240501 20:09:29.786639 31278 tablet_copy_service.cc:433] P 341d3bb73a5c4e8f96176a281db4a60f: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-a3b0a8894c0f46c683461a1857ff27ff on tablet a3b0a8894c0f46c683461a1857ff27ff with peer c42de85026a84ed7b4f462cf25433e01
I20240501 20:09:29.855125 31644 consensus_queue.cc:237] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:29.858114 31777 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 20:09:29.858767 1013 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240501 20:09:29.861665 31908 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 20:09:29.862130 1013 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240501 20:09:29.865262 1013 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index 96 to 98, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.866477 31908 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 96 to 98, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.870648 31777 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 96 to 98, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.874608 31113 catalog_manager.cc:5484] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 96 to 98, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:29.888409 407 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 20:09:29.891218 31385 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 97. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240501 20:09:29.891305 955 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 96, Time since last communication: 0.000s
I20240501 20:09:29.894140 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e93ddb68383c43089544a6f0549c83e2 with cas_config_opid_index 96: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:29.896767 1064 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20240501 20:09:29.896834 406 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 96 to 98, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:29.910449 31385 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index 96 to 98, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.113955 31498 tablet_service.cc:1514] Processing DeleteTablet for tablet e93ddb68383c43089544a6f0549c83e2 with delete_type TABLET_DATA_TOMBSTONED (TS ec65c12cd0724cf9aa16fcd61bbedae4 not found in new config with opid_index 98) from {username='slave'} at 127.0.0.1:56620
I20240501 20:09:30.122661 1224 tablet_replica.cc:330] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: stopping tablet replica
I20240501 20:09:30.136574 1224 raft_consensus.cc:2229] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:30.137044 1224 raft_consensus.cc:2258] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:30.139184 1224 ts_tablet_manager.cc:1875] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:30.160255 406 raft_consensus.cc:1203] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.102->[1.103-1.116] Dedup: 1.112->[1.113-1.116]
I20240501 20:09:30.191494 1224 ts_tablet_manager.cc:1888] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.97
I20240501 20:09:30.191867 1224 log.cc:1196] T e93ddb68383c43089544a6f0549c83e2 P ec65c12cd0724cf9aa16fcd61bbedae4: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/e93ddb68383c43089544a6f0549c83e2
I20240501 20:09:30.220336 31102 catalog_manager.cc:4830] TS ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): tablet e93ddb68383c43089544a6f0549c83e2 (table rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]) successfully deleted
I20240501 20:09:30.308109 1208 raft_consensus.cc:1050] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:30.309300 1208 consensus_queue.cc:237] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:30.313895 405 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:30.315323 1161 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.001s
I20240501 20:09:30.317409 31777 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:30.317991 1176 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240501 20:09:30.326534 31517 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:30.327607 1176 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240501 20:09:30.334535 31386 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:30.335219 1067 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 115, Time since last communication: 0.000s
I20240501 20:09:30.342525 31517 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 103 to 118, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.342391 1161 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index 103 to 118, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.346984 31386 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 103 to 118, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.377687 31649 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:30.402444 31115 catalog_manager.cc:5484] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: config changed from index 103 to 118, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.408921 1176 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240501 20:09:30.412381 405 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 103 to 118, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.426621 31642 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 103 to 118, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:30.473557 31192 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 62896.7us
I20240501 20:09:30.474242 31777 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 103 to 118, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:30.533828 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 114841us
I20240501 20:09:30.554579 31254 consensus_queue.cc:237] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 1.119, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:30.557967 407 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240501 20:09:30.558701 1161 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 118, Time since last communication: 0.000s
I20240501 20:09:30.560261 31388 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240501 20:09:30.560918 1244 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 118, Time since last communication: 0.000s
I20240501 20:09:30.561084 407 raft_consensus.cc:1203] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.118->[1.119-1.120] Dedup: 1.119->[1.120-1.120]
I20240501 20:09:30.566419 1161 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.120: config changed from index 118 to 120, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.572585 31104 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3b0a8894c0f46c683461a1857ff27ff with cas_config_opid_index 118: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:30.573611 407 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 118 to 120, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.575760 31115 catalog_manager.cc:5484] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index 118 to 120, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:30.586261 31759 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b0a8894c0f46c683461a1857ff27ff with delete_type TABLET_DATA_TOMBSTONED (TS d73d71e1531b4649b363236998019221 not found in new config with opid_index 120) from {username='slave'} at 127.0.0.1:37186
I20240501 20:09:30.603893 31513 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240501 20:09:30.604475 31386 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 118 to 120, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.608453 1244 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 118, Time since last communication: 0.000s
I20240501 20:09:30.609934 31513 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 118 to 120, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.619138 31642 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 120. (index mismatch)
I20240501 20:09:30.646370 1253 tablet_replica.cc:330] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: stopping tablet replica
I20240501 20:09:30.677129 1244 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 120, Last known committed idx: 118, Time since last communication: 0.000s
I20240501 20:09:30.704480 1253 raft_consensus.cc:2229] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:30.704924 1253 pending_rounds.cc:62] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: Trying to abort 1 pending ops.
I20240501 20:09:30.705173 1253 pending_rounds.cc:68] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: Aborting op as it isn't in flight: id { term: 1 index: 119 } timestamp: 7022977722558115840 op_type: WRITE_OP write_request { tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\310\267\2373\r\261\371p\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""6673756c4c443259" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977722224926720 authz_token { token_data: "\010\223\305\312\261\006\"3\n\005slave\022*\n ab3c75c3bb2748fcb14f718749a6bfbb\020\001\030\001 \001(\001" signature: "<redacted>""\317\022U\376\005d\255\203\3431f\247\033u\007\327=\262f\244\375\367\250\014\247\376\253 y\244\2439=w\365\260\323\226\004 \371\245\213-K4ba\360\363iZ\251x9CB\251C\223\256\232>@" signing_key_seq_num: 0 } } request_id { client_id: "aa9472d6542d4f98ae89b0b9f36e963a" seq_no: 315 first_incomplete_seq_no: 315 attempt_no: 0 }
I20240501 20:09:30.705945 1253 raft_consensus.cc:2258] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:30.752058 1253 ts_tablet_manager.cc:1875] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:30.752856 31642 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.120: config changed from index 118 to 120, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 120 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:30.843835 1253 ts_tablet_manager.cc:1888] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.119
I20240501 20:09:30.844041 1253 log.cc:1196] T a3b0a8894c0f46c683461a1857ff27ff P d73d71e1531b4649b363236998019221: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/wals/a3b0a8894c0f46c683461a1857ff27ff
I20240501 20:09:30.848345 31102 catalog_manager.cc:4830] TS d73d71e1531b4649b363236998019221 (127.28.136.69:43505): tablet a3b0a8894c0f46c683461a1857ff27ff (table rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]) successfully deleted
I20240501 20:09:31.648154 31649 consensus_queue.cc:237] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:31.652205 31775 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 20:09:31.653538 31906 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 20:09:31.654320 1216 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 20:09:31.657680 1216 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 20:09:31.660856 1013 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:31.665983 31780 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:31.669055 31111 catalog_manager.cc:5484] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index -1 to 96, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:31.669049 31908 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:31.701038 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 87516.2us
I20240501 20:09:31.776381 31255 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 20:09:31.776839 31388 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
W20240501 20:09:31.787039 31585 consensus_peers.cc:485] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dda27412e67c4571bd8c5e86a65b787f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:31.787920 1064 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 20:09:31.795183 852 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 20:09:31.796571 31388 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:31.797827 31254 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:32.415050 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 57414us
I20240501 20:09:32.455849 1290 ts_tablet_manager.cc:902] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:09:32.673288 31513 consensus_queue.cc:237] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.232, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:32.679096 31780 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240501 20:09:32.679771 1053 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 231, Time since last communication: 0.000s
I20240501 20:09:32.691732 31908 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240501 20:09:32.695503 1053 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 231, Time since last communication: 0.000s
I20240501 20:09:32.725153 735 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.233: config changed from index -1 to 233, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:32.726550 31904 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index -1 to 233, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:32.746752 31777 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index -1 to 233, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:32.751957 31115 catalog_manager.cc:5484] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index -1 to 233, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:32.818450 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 166522us
I20240501 20:09:32.820348 31256 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240501 20:09:32.823586 31385 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 232. Preceding OpId from leader: term: 1 index: 233. (index mismatch)
I20240501 20:09:32.826401 1220 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 231, Time since last communication: 0.000s
I20240501 20:09:32.860430 1290 tablet_copy_client.cc:308] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:44049
W20240501 20:09:32.863624 31455 consensus_peers.cc:485] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abdd2df68c774be1ab6e39d9d5dbf563. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:32.866099 1220 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 233, Last known committed idx: 230, Time since last communication: 0.000s
I20240501 20:09:32.872148 31385 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index -1 to 233, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:32.876421 31255 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.233: config changed from index -1 to 233, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:32.889912 31669 tablet_copy_service.cc:140] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Received BeginTabletCopySession request for tablet dda27412e67c4571bd8c5e86a65b787f from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:49171)
I20240501 20:09:32.891604 31669 tablet_copy_service.cc:161] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Beginning new tablet copy session on tablet dda27412e67c4571bd8c5e86a65b787f from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:49171: session id = c42de85026a84ed7b4f462cf25433e01-dda27412e67c4571bd8c5e86a65b787f
I20240501 20:09:32.894927 31669 tablet_copy_source_session.cc:205] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:32.901955 1290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda27412e67c4571bd8c5e86a65b787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:33.016129 1290 tablet_copy_client.cc:783] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:33.016562 1290 tablet_copy_client.cc:647] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:33.162746 1290 tablet_copy_client.cc:523] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:33.178557 1290 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
I20240501 20:09:33.261443 1331 ts_tablet_manager.cc:902] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
W20240501 20:09:33.265333 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 127431us
I20240501 20:09:33.292431 1331 tablet_copy_client.cc:308] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:42361
I20240501 20:09:33.293917 31539 tablet_copy_service.cc:140] P ec65c12cd0724cf9aa16fcd61bbedae4: Received BeginTabletCopySession request for tablet abdd2df68c774be1ab6e39d9d5dbf563 from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:45421)
I20240501 20:09:33.294118 31539 tablet_copy_service.cc:161] P ec65c12cd0724cf9aa16fcd61bbedae4: Beginning new tablet copy session on tablet abdd2df68c774be1ab6e39d9d5dbf563 from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:45421: session id = c42de85026a84ed7b4f462cf25433e01-abdd2df68c774be1ab6e39d9d5dbf563
I20240501 20:09:33.296443 31539 tablet_copy_source_session.cc:205] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:33.297771 1331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdd2df68c774be1ab6e39d9d5dbf563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:33.308776 1331 tablet_copy_client.cc:783] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:33.316999 1331 tablet_copy_client.cc:647] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:33.325496 1331 tablet_copy_client.cc:523] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:33.333550 1331 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
W20240501 20:09:33.382872 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 54673us
I20240501 20:09:33.545794 1290 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=107 overwritten=0 applied=107 ignored=0} inserts{seen=779 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:33.546283 1290 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:33.546624 1290 ts_tablet_manager.cc:1367] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 0.368s user 0.159s sys 0.005s
I20240501 20:09:33.547437 1290 raft_consensus.cc:348] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:33.547798 1290 raft_consensus.cc:729] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:33.548030 1290 consensus_queue.cc:260] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
W20240501 20:09:33.561542 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 159249us
I20240501 20:09:33.569283 1290 ts_tablet_manager.cc:1398] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20240501 20:09:33.643920 31669 tablet_copy_service.cc:342] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-dda27412e67c4571bd8c5e86a65b787f received from {username='slave'} at 127.28.136.71:49171
I20240501 20:09:33.644073 31669 tablet_copy_service.cc:433] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-dda27412e67c4571bd8c5e86a65b787f on tablet dda27412e67c4571bd8c5e86a65b787f with peer c42de85026a84ed7b4f462cf25433e01
I20240501 20:09:33.856806 406 raft_consensus.cc:1203] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.95->[1.96-1.110] Dedup: 1.107->[1.108-1.110]
I20240501 20:09:33.917383 1331 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=238 overwritten=0 applied=238 ignored=0} inserts{seen=869 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:33.917840 1331 tablet_bootstrap.cc:492] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:33.918157 1331 ts_tablet_manager.cc:1367] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 0.585s user 0.289s sys 0.017s
I20240501 20:09:33.918879 1331 raft_consensus.cc:348] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:33.919166 1331 raft_consensus.cc:729] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:33.919390 1331 consensus_queue.cc:260] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 238, Last appended: 1.238, Last appended by leader: 238, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 233 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:33.920070 1331 ts_tablet_manager.cc:1398] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:33.923287 31539 tablet_copy_service.cc:342] P ec65c12cd0724cf9aa16fcd61bbedae4: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-abdd2df68c774be1ab6e39d9d5dbf563 received from {username='slave'} at 127.28.136.71:45421
I20240501 20:09:33.923437 31539 tablet_copy_service.cc:433] P ec65c12cd0724cf9aa16fcd61bbedae4: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-abdd2df68c774be1ab6e39d9d5dbf563 on tablet abdd2df68c774be1ab6e39d9d5dbf563 with peer c42de85026a84ed7b4f462cf25433e01
W20240501 20:09:34.052363 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 104953us
I20240501 20:09:34.216156 1353 raft_consensus.cc:1050] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:34.217262 1353 consensus_queue.cc:237] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 1.111, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:34.220499 31775 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240501 20:09:34.221664 407 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240501 20:09:34.222147 31388 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240501 20:09:34.224079 31908 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240501 20:09:34.226714 852 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 20:09:34.227334 1284 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 20:09:34.228127 1284 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 20:09:34.247287 31252 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240501 20:09:34.258388 1353 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 20:09:34.272589 1353 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s
I20240501 20:09:34.274820 1370 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.112: config changed from index 96 to 112, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.279664 31773 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 96 to 112, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.288650 31908 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 96 to 112, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.290666 31115 catalog_manager.cc:5484] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 reported cstate change: config changed from index 96 to 112, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.316391 404 raft_consensus.cc:1203] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.232->[1.233-1.243] Dedup: 1.238->[1.239-1.243]
I20240501 20:09:34.348091 406 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 96 to 112, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.353335 31256 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 96 to 112, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.369488 31385 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 96 to 112, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:34.450536 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 46094us
I20240501 20:09:34.494560 1053 raft_consensus.cc:1050] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:34.495447 1053 consensus_queue.cc:237] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 1.246, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:34.499439 407 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240501 20:09:34.500105 1275 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 244, Time since last communication: 0.000s
I20240501 20:09:34.502765 31254 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240501 20:09:34.503394 1275 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 245, Time since last communication: 0.000s
I20240501 20:09:34.512845 404 raft_consensus.cc:1203] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.245->[1.246-1.246] Dedup: 1.246->[]
I20240501 20:09:34.523805 31904 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240501 20:09:34.524441 1365 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 245, Time since last communication: 0.004s
W20240501 20:09:34.532321 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 55201.6us
I20240501 20:09:34.541749 1275 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.247: config changed from index 233 to 247, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.547288 31775 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240501 20:09:34.551530 407 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 233 to 247, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.551602 31254 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 233 to 247, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.555012 1053 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 243, Time since last communication: 0.000s
I20240501 20:09:34.567175 31780 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 233 to 247, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.570878 31909 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 233 to 247, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.592366 31386 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 244. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240501 20:09:34.616705 1053 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 244, Time since last communication: 0.000s
I20240501 20:09:34.629825 31386 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index 233 to 247, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:34.642130 31715 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 54570.6us
I20240501 20:09:34.906703 31109 catalog_manager.cc:5484] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 reported cstate change: config changed from index 233 to 247, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:34.930161 31644 consensus_queue.cc:237] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:34.933153 31905 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:34.933786 1216 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240501 20:09:34.936709 31254 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:34.942744 407 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:34.952913 1216 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240501 20:09:34.966420 1216 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240501 20:09:34.974613 31774 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 20:09:34.976506 852 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20240501 20:09:34.976189 1216 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index 112 to 118, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.978614 31254 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 112 to 118, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.981415 31774 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 112 to 118, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:34.998494 404 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 112 to 118, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:35.006306 31908 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 112 to 118, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:35.025434 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dda27412e67c4571bd8c5e86a65b787f with cas_config_opid_index 112: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:35.047540 31111 catalog_manager.cc:5484] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index 112 to 118, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:35.065353 31513 consensus_queue.cc:237] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:35.069733 31906 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 20:09:35.070494 1053 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 20:09:35.078055 407 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 20:09:35.078675 1275 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 20:09:35.082870 31386 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 20:09:35.085572 31367 tablet_service.cc:1514] Processing DeleteTablet for tablet dda27412e67c4571bd8c5e86a65b787f with delete_type TABLET_DATA_TOMBSTONED (TS db07c3a323eb4a3c919eaf9b19b24986 not found in new config with opid_index 118) from {username='slave'} at 127.0.0.1:51676
I20240501 20:09:35.085551 1275 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 20:09:35.094429 1399 tablet_replica.cc:330] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: stopping tablet replica
I20240501 20:09:35.094873 1399 raft_consensus.cc:2229] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:35.095679 1399 raft_consensus.cc:2258] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:35.097647 735 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index 247 to 253, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:35.101958 1399 ts_tablet_manager.cc:1875] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:35.104502 31114 catalog_manager.cc:5484] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 reported cstate change: config changed from index 247 to 253, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:35.105219 31102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet abdd2df68c774be1ab6e39d9d5dbf563 with cas_config_opid_index 247: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:35.104449 31906 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 247 to 253, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:35.105335 405 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 247 to 253, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:35.108834 31237 tablet_service.cc:1514] Processing DeleteTablet for tablet abdd2df68c774be1ab6e39d9d5dbf563 with delete_type TABLET_DATA_TOMBSTONED (TS 341d3bb73a5c4e8f96176a281db4a60f not found in new config with opid_index 253) from {username='slave'} at 127.0.0.1:51884
I20240501 20:09:35.110357 31779 raft_consensus.cc:1261] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240501 20:09:35.110994 1275 consensus_queue.cc:1035] T abdd2df68c774be1ab6e39d9d5dbf563 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240501 20:09:35.114794 31779 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 247 to 253, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:35.132578 1399 ts_tablet_manager.cc:1888] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.117
I20240501 20:09:35.144891 1399 log.cc:1196] T dda27412e67c4571bd8c5e86a65b787f P db07c3a323eb4a3c919eaf9b19b24986: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/dda27412e67c4571bd8c5e86a65b787f
I20240501 20:09:35.155717 31104 catalog_manager.cc:4830] TS db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): tablet dda27412e67c4571bd8c5e86a65b787f (table rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]) successfully deleted
I20240501 20:09:35.156183 31385 raft_consensus.cc:2932] T abdd2df68c774be1ab6e39d9d5dbf563 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 247 to 253, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:35.157624 31323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 178430us
I20240501 20:09:35.190618 1400 tablet_replica.cc:330] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: stopping tablet replica
I20240501 20:09:35.231504 1400 raft_consensus.cc:2229] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:35.231899 1400 raft_consensus.cc:2258] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:35.233212 1400 ts_tablet_manager.cc:1875] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:35.267499 1400 ts_tablet_manager.cc:1888] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.252
I20240501 20:09:35.267668 1400 log.cc:1196] T abdd2df68c774be1ab6e39d9d5dbf563 P 341d3bb73a5c4e8f96176a281db4a60f: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/abdd2df68c774be1ab6e39d9d5dbf563
I20240501 20:09:35.268695 31104 catalog_manager.cc:4830] TS 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): tablet abdd2df68c774be1ab6e39d9d5dbf563 (table rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]) successfully deleted
W20240501 20:09:35.315062 31323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 62322.8us
W20240501 20:09:35.512688 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 49770.7us
W20240501 20:09:35.810091 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 55680.3us
I20240501 20:09:36.242739 31251 consensus_queue.cc:237] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.248, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
W20240501 20:09:36.249166 31194 consensus_peers.cc:485] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afb53b647def417c8c9e2378dbeb5dd9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:36.257182 31780 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240501 20:09:36.261261 1244 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 247, Time since last communication: 0.000s
I20240501 20:09:36.263372 31519 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240501 20:09:36.270517 1430 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 247, Time since last communication: 0.000s
I20240501 20:09:36.289791 31779 raft_consensus.cc:1203] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.247->[1.248-1.248] Dedup: 1.248->[]
I20240501 20:09:36.297263 1423 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.249: config changed from index -1 to 249, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.312224 31514 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index -1 to 249, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.317082 31114 catalog_manager.cc:5484] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index -1 to 249, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:36.338192 31775 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index -1 to 249, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.368634 31644 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240501 20:09:36.464054 31904 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240501 20:09:36.465106 1161 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 247, Time since last communication: 0.000s
I20240501 20:09:36.471065 31908 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index -1 to 249, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.472105 1161 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 247, Time since last communication: 0.000s
W20240501 20:09:36.574761 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 197881us
I20240501 20:09:36.577518 31647 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.249: config changed from index -1 to 249, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.674055 31514 consensus_queue.cc:237] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 1.200, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:36.677287 31387 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 200. Preceding OpId from leader: term: 1 index: 201. (index mismatch)
I20240501 20:09:36.677409 31254 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 200. Preceding OpId from leader: term: 1 index: 201. (index mismatch)
I20240501 20:09:36.678026 1439 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240501 20:09:36.679872 31647 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 200. Preceding OpId from leader: term: 1 index: 201. (index mismatch)
I20240501 20:09:36.680895 31904 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 200. Preceding OpId from leader: term: 1 index: 201. (index mismatch)
I20240501 20:09:36.682502 1439 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240501 20:09:36.685736 1439 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.201: config changed from index -1 to 201, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.688939 1275 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240501 20:09:36.689240 31387 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.201: config changed from index -1 to 201, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.689800 1275 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240501 20:09:36.694633 31647 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.201: config changed from index -1 to 201, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:36.697731 31455 consensus_peers.cc:485] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37fbd40e4b9a4d75994b3e138c4506d8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:36.698525 31109 catalog_manager.cc:5484] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: config changed from index -1 to 201, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.701038 31904 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.201: config changed from index -1 to 201, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.710083 31252 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.201: config changed from index -1 to 201, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:36.807442 1466 ts_tablet_manager.cc:902] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401)
I20240501 20:09:37.225991 1466 tablet_copy_client.cc:308] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:33401
I20240501 20:09:37.227121 31278 tablet_copy_service.cc:140] P 341d3bb73a5c4e8f96176a281db4a60f: Received BeginTabletCopySession request for tablet afb53b647def417c8c9e2378dbeb5dd9 from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:47593)
I20240501 20:09:37.227314 31278 tablet_copy_service.cc:161] P 341d3bb73a5c4e8f96176a281db4a60f: Beginning new tablet copy session on tablet afb53b647def417c8c9e2378dbeb5dd9 from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:47593: session id = c42de85026a84ed7b4f462cf25433e01-afb53b647def417c8c9e2378dbeb5dd9
I20240501 20:09:37.229983 31278 tablet_copy_source_session.cc:205] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:37.231340 1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb53b647def417c8c9e2378dbeb5dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.238186 1466 tablet_copy_client.cc:783] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:37.238539 1466 tablet_copy_client.cc:647] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:37.241214 1466 tablet_copy_client.cc:523] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:37.255630 1466 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
W20240501 20:09:37.255702 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 46727.5us
I20240501 20:09:37.373867 31644 consensus_queue.cc:237] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 1.137, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:37.384263 31908 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:37.385149 1013 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:37.415196 31385 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:37.422322 31515 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:37.427978 1479 ts_tablet_manager.cc:902] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:09:37.432539 31251 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch)
I20240501 20:09:37.432422 1013 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:37.446442 1013 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.138: config changed from index -1 to 138, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:37.448088 31514 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index -1 to 138, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:37.478893 31109 catalog_manager.cc:5484] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 reported cstate change: config changed from index -1 to 138, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
W20240501 20:09:37.545253 31582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 74024.6us
W20240501 20:09:37.546757 31583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 75513.4us
I20240501 20:09:37.549525 1414 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:37.550616 1414 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 20:09:37.580096 1479 tablet_copy_client.cc:308] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:42361
I20240501 20:09:37.581776 31539 tablet_copy_service.cc:140] P ec65c12cd0724cf9aa16fcd61bbedae4: Received BeginTabletCopySession request for tablet 37fbd40e4b9a4d75994b3e138c4506d8 from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:45421)
I20240501 20:09:37.581947 31539 tablet_copy_service.cc:161] P ec65c12cd0724cf9aa16fcd61bbedae4: Beginning new tablet copy session on tablet 37fbd40e4b9a4d75994b3e138c4506d8 from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:45421: session id = c42de85026a84ed7b4f462cf25433e01-37fbd40e4b9a4d75994b3e138c4506d8
I20240501 20:09:37.584631 31539 tablet_copy_source_session.cc:205] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:37.641284 31904 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index -1 to 138, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:37.706238 1479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd40e4b9a4d75994b3e138c4506d8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:09:37.852932 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 304360us
W20240501 20:09:37.853410 31585 consensus_peers.cc:485] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfe38cf1f2ef48e4adca9910e82e2d9e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:37.856474 31385 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index -1 to 138, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:37.860412 31255 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index -1 to 138, NON_VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } } }
I20240501 20:09:37.899768 1479 tablet_copy_client.cc:783] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:37.900048 1479 tablet_copy_client.cc:647] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:37.911350 1479 tablet_copy_client.cc:523] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:37.925474 1479 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
I20240501 20:09:37.991998 1508 ts_tablet_manager.cc:902] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: Initiating tablet copy from peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
W20240501 20:09:38.085000 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 70536us
I20240501 20:09:38.326105 1508 tablet_copy_client.cc:308] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:44049
I20240501 20:09:38.346324 31669 tablet_copy_service.cc:140] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Received BeginTabletCopySession request for tablet dfe38cf1f2ef48e4adca9910e82e2d9e from peer c42de85026a84ed7b4f462cf25433e01 ({username='slave'} at 127.28.136.71:49171)
I20240501 20:09:38.346530 31669 tablet_copy_service.cc:161] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Beginning new tablet copy session on tablet dfe38cf1f2ef48e4adca9910e82e2d9e from peer c42de85026a84ed7b4f462cf25433e01 at {username='slave'} at 127.28.136.71:49171: session id = c42de85026a84ed7b4f462cf25433e01-dfe38cf1f2ef48e4adca9910e82e2d9e
I20240501 20:09:38.350340 31669 tablet_copy_source_session.cc:205] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:38.364557 1508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe38cf1f2ef48e4adca9910e82e2d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:38.376466 1508 tablet_copy_client.cc:783] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:38.376883 1508 tablet_copy_client.cc:647] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:38.416471 1508 tablet_copy_client.cc:523] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:38.420132 1508 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: Bootstrap starting.
W20240501 20:09:38.507998 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 62979.7us
W20240501 20:09:38.575076 1466 log.cc:924] Time spent T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: Append to log took a long time: real 0.144s user 0.003s sys 0.000s
W20240501 20:09:38.660187 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 67726.4us
W20240501 20:09:38.660841 1466 log.cc:924] Time spent T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
W20240501 20:09:38.660682 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 66751.2us
W20240501 20:09:38.853255 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 81104.1us
W20240501 20:09:39.092119 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 62127.5us
W20240501 20:09:39.175144 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 81574.6us
W20240501 20:09:39.260653 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 45174.4us
W20240501 20:09:39.518539 31194 consensus_peers.cc:485] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 20:09:39.572906 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 93958.6us
W20240501 20:09:39.807400 31582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 97883.7us
W20240501 20:09:39.953047 31255 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.062s sys 0.067s
W20240501 20:09:39.954247 31255 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.744s user 0.062s sys 0.067s
W20240501 20:09:40.054672 31582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 63071.4us
W20240501 20:09:40.153311 1479 log.cc:924] Time spent T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
W20240501 20:09:40.341220 31585 consensus_peers.cc:485] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): Couldn't send request to peer c42de85026a84ed7b4f462cf25433e01. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 20:09:40.386581 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 93564.8us
I20240501 20:09:40.415092 1479 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=210 overwritten=0 applied=210 ignored=0} inserts{seen=895 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:40.417204 1479 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:40.419956 1479 ts_tablet_manager.cc:1367] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 2.495s user 0.264s sys 0.025s
I20240501 20:09:40.421005 1479 raft_consensus.cc:348] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:40.429119 1479 raft_consensus.cc:729] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:40.429530 1479 consensus_queue.cc:260] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 210, Last appended: 1.210, Last appended by leader: 210, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:40.444340 1466 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=259 overwritten=0 applied=258 ignored=0} inserts{seen=930 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:09:40.448338 1466 tablet_bootstrap.cc:492] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:40.451496 1466 ts_tablet_manager.cc:1367] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 3.205s user 0.341s sys 0.040s
I20240501 20:09:40.453012 1466 raft_consensus.cc:348] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:40.483604 1466 raft_consensus.cc:729] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:40.484680 1466 consensus_queue.cc:260] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 258, Last appended: 1.259, Last appended by leader: 259, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 249 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:40.505501 1466 ts_tablet_manager.cc:1398] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.053s user 0.004s sys 0.000s
I20240501 20:09:40.530058 1479 ts_tablet_manager.cc:1398] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.110s user 0.010s sys 0.016s
I20240501 20:09:40.533516 31539 tablet_copy_service.cc:342] P ec65c12cd0724cf9aa16fcd61bbedae4: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-37fbd40e4b9a4d75994b3e138c4506d8 received from {username='slave'} at 127.28.136.71:45421
I20240501 20:09:40.533681 31539 tablet_copy_service.cc:433] P ec65c12cd0724cf9aa16fcd61bbedae4: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-37fbd40e4b9a4d75994b3e138c4506d8 on tablet 37fbd40e4b9a4d75994b3e138c4506d8 with peer c42de85026a84ed7b4f462cf25433e01
W20240501 20:09:40.554500 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 75267.4us
I20240501 20:09:40.585405 407 raft_consensus.cc:1203] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.200->[1.201-1.233] Dedup: 1.210->[1.211-1.233]
W20240501 20:09:40.629455 31191 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 392747us
W20240501 20:09:40.630870 1599 log.cc:924] Time spent T 2c9d9115e89c437ca7b962201115f087 P 341d3bb73a5c4e8f96176a281db4a60f: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20240501 20:09:40.632462 31278 tablet_copy_service.cc:342] P 341d3bb73a5c4e8f96176a281db4a60f: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-afb53b647def417c8c9e2378dbeb5dd9 received from {username='slave'} at 127.28.136.71:47593
I20240501 20:09:40.632582 31278 tablet_copy_service.cc:433] P 341d3bb73a5c4e8f96176a281db4a60f: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-afb53b647def417c8c9e2378dbeb5dd9 on tablet afb53b647def417c8c9e2378dbeb5dd9 with peer c42de85026a84ed7b4f462cf25433e01
I20240501 20:09:40.710652 405 raft_consensus.cc:1203] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.248->[1.249-1.293] Dedup: 1.259->[1.260-1.293]
W20240501 20:09:40.843072 1508 log.cc:924] Time spent T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: Append to log took a long time: real 0.282s user 0.000s sys 0.000s
I20240501 20:09:40.850952 1527 raft_consensus.cc:1050] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:40.862691 1527 consensus_queue.cc:237] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:40.877883 405 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:09:40.878453 31515 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:09:40.878472 31779 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:09:40.878541 1592 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:09:40.879092 1588 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:09:40.880457 1588 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:09:40.889386 1596 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.294: config changed from index 249 to 294, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:40.889647 405 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 249 to 294, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
W20240501 20:09:40.895066 31191 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 203074us
I20240501 20:09:40.902801 31775 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 249 to 294, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:40.908324 31109 catalog_manager.cc:5484] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 reported cstate change: config changed from index 249 to 294, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:40.908174 31513 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 249 to 294, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:40.922838 31904 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:09:40.939726 1596 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:09:40.951432 31644 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:09:40.988710 1527 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:09:40.961881 31904 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 249 to 294, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.044464 31644 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 249 to 294, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.277968 1520 raft_consensus.cc:1050] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:41.279009 1520 consensus_queue.cc:237] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 1.238, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:41.282532 31647 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240501 20:09:41.283064 1452 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240501 20:09:41.289124 31258 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240501 20:09:41.289716 1220 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240501 20:09:41.290313 31904 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240501 20:09:41.290835 1220 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240501 20:09:41.315475 404 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240501 20:09:41.316584 1452 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240501 20:09:41.329262 31389 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240501 20:09:41.329828 31904 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 201 to 239, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.333905 1602 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240501 20:09:41.311414 1220 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.239: config changed from index 201 to 239, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.347004 31111 catalog_manager.cc:5484] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 201 to 239, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.350915 31252 consensus_queue.cc:237] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:41.351964 404 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 201 to 239, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.362112 31774 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch)
I20240501 20:09:41.364615 1596 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240501 20:09:41.365410 31644 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch)
I20240501 20:09:41.366526 1596 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240501 20:09:41.367063 406 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch)
I20240501 20:09:41.370994 1067 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240501 20:09:41.388028 1527 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.297: config changed from index 294 to 297, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.395093 31111 catalog_manager.cc:5484] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index 294 to 297, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.395923 31774 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 294 to 297, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.399055 31888 tablet_service.cc:1514] Processing DeleteTablet for tablet afb53b647def417c8c9e2378dbeb5dd9 with delete_type TABLET_DATA_TOMBSTONED (TS 582c7a51c0ac43b4934078ebdf434485 not found in new config with opid_index 297) from {username='slave'} at 127.0.0.1:46678
I20240501 20:09:41.388824 31388 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 201 to 239, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.413231 31104 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet afb53b647def417c8c9e2378dbeb5dd9 with cas_config_opid_index 294: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:41.414069 407 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 294 to 297, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.416090 31647 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 294 to 297, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.439620 31515 raft_consensus.cc:1261] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch)
I20240501 20:09:41.466568 1527 consensus_queue.cc:1035] T afb53b647def417c8c9e2378dbeb5dd9 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240501 20:09:41.468802 31515 raft_consensus.cc:2932] T afb53b647def417c8c9e2378dbeb5dd9 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 294 to 297, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.534586 1642 tablet_replica.cc:330] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: stopping tablet replica
I20240501 20:09:41.534897 1642 raft_consensus.cc:2229] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:41.554106 1642 raft_consensus.cc:2258] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:41.569753 1642 ts_tablet_manager.cc:1875] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:41.585516 1642 ts_tablet_manager.cc:1888] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.296
I20240501 20:09:41.585852 1642 log.cc:1196] T afb53b647def417c8c9e2378dbeb5dd9 P 582c7a51c0ac43b4934078ebdf434485: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/wals/afb53b647def417c8c9e2378dbeb5dd9
I20240501 20:09:41.588212 31101 catalog_manager.cc:4830] TS 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): tablet afb53b647def417c8c9e2378dbeb5dd9 (table rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]) successfully deleted
I20240501 20:09:41.595134 31647 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 201 to 239, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.605240 31258 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 201 to 239, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.623103 31519 consensus_queue.cc:237] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 1.240, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:41.631253 31388 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20240501 20:09:41.632498 31909 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20240501 20:09:41.641464 31644 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20240501 20:09:41.642277 1220 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 239, Time since last communication: 0.000s
I20240501 20:09:41.642850 1220 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 239, Time since last communication: 0.000s
I20240501 20:09:41.643630 1602 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 239, Time since last communication: 0.000s
I20240501 20:09:41.646153 1602 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.241: config changed from index 239 to 241, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.647729 31643 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 239 to 241, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.659211 31111 catalog_manager.cc:5484] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 239 to 241, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.662676 404 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch)
I20240501 20:09:41.667250 31237 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fbd40e4b9a4d75994b3e138c4506d8 with delete_type TABLET_DATA_TOMBSTONED (TS 341d3bb73a5c4e8f96176a281db4a60f not found in new config with opid_index 241) from {username='slave'} at 127.0.0.1:51884
I20240501 20:09:41.672806 1650 tablet_replica.cc:330] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: stopping tablet replica
I20240501 20:09:41.673293 1650 raft_consensus.cc:2229] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:41.673974 1650 pending_rounds.cc:62] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: Trying to abort 1 pending ops.
I20240501 20:09:41.674109 1650 pending_rounds.cc:68] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: Aborting op as it isn't in flight: id { term: 1 index: 240 } timestamp: 7022977767165861888 op_type: WRITE_OP write_request { tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\225\006\275B\332G\306\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""eba38e887dc53233" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977766744862720 authz_token { token_data: "\010\224\305\312\261\006\"3\n\005slave\022*\n 0cec196ed65049adb8e6a6690675a13e\020\001\030\001 \001(\001" signature: "<redacted>""\212\002\205\023K\331\374<\2621\3245\313WM \362?\254\364Meoo\\\371\326h\315S>\222H[e\211|\347\355\240f[\360\222\373\201\3228\332\035\342\217\312\246a\314w2\000k)\177e\316" signing_key_seq_num: 0 } } request_id { client_id: "cee00185e9f5493fac1b72a466a385f8" seq_no: 730 first_incomplete_seq_no: 730 attempt_no: 0 }
I20240501 20:09:41.674485 1520 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 239, Time since last communication: 0.000s
I20240501 20:09:41.674930 1650 raft_consensus.cc:2258] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:41.676141 31102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37fbd40e4b9a4d75994b3e138c4506d8 with cas_config_opid_index 239: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:41.679311 31388 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 239 to 241, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.680617 404 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 239 to 241, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.682966 31909 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 239 to 241, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:41.702337 1650 ts_tablet_manager.cc:1875] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:41.969530 1508 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: Bootstrap replayed 1/1 log segments. Stats: ops{read=143 overwritten=0 applied=143 ignored=0} inserts{seen=828 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:41.993296 1508 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: Bootstrap complete.
I20240501 20:09:41.994465 1508 ts_tablet_manager.cc:1367] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: Time spent bootstrapping tablet: real 3.574s user 0.208s sys 0.012s
I20240501 20:09:41.996208 1508 raft_consensus.cc:348] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:41.997925 1508 raft_consensus.cc:729] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42de85026a84ed7b4f462cf25433e01, State: Initialized, Role: LEARNER
I20240501 20:09:41.998374 1508 consensus_queue.cc:260] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 143, Last appended: 1.143, Last appended by leader: 143, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: true } }
I20240501 20:09:41.999471 1508 ts_tablet_manager.cc:1398] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:42.085052 1650 ts_tablet_manager.cc:1888] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.240
I20240501 20:09:42.087548 1650 log.cc:1196] T 37fbd40e4b9a4d75994b3e138c4506d8 P 341d3bb73a5c4e8f96176a281db4a60f: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/37fbd40e4b9a4d75994b3e138c4506d8
I20240501 20:09:42.088438 31104 catalog_manager.cc:4830] TS 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): tablet 37fbd40e4b9a4d75994b3e138c4506d8 (table rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]) successfully deleted
I20240501 20:09:42.125445 31669 tablet_copy_service.cc:342] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Request end of tablet copy session c42de85026a84ed7b4f462cf25433e01-dfe38cf1f2ef48e4adca9910e82e2d9e received from {username='slave'} at 127.28.136.71:49171
I20240501 20:09:42.125630 31669 tablet_copy_service.cc:433] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: ending tablet copy session c42de85026a84ed7b4f462cf25433e01-dfe38cf1f2ef48e4adca9910e82e2d9e on tablet dfe38cf1f2ef48e4adca9910e82e2d9e with peer c42de85026a84ed7b4f462cf25433e01
W20240501 20:09:42.356575 31843 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 44899.5us
I20240501 20:09:42.463121 406 raft_consensus.cc:1203] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.137->[1.138-1.153] Dedup: 1.143->[1.144-1.153]
W20240501 20:09:42.524533 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 75337.3us
W20240501 20:09:42.749552 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 72065.5us
I20240501 20:09:42.876647 1682 raft_consensus.cc:1050] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: attempting to promote NON_VOTER c42de85026a84ed7b4f462cf25433e01 to VOTER
I20240501 20:09:42.878357 1682 consensus_queue.cc:237] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 153, Committed index: 153, Last appended: 1.153, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:42.882433 31515 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240501 20:09:42.883069 1638 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240501 20:09:42.888443 31906 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240501 20:09:42.891222 1581 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240501 20:09:42.908082 403 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 LEARNER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240501 20:09:42.914458 31254 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240501 20:09:42.945339 31385 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 153. Preceding OpId from leader: term: 1 index: 154. (index mismatch)
I20240501 20:09:42.999066 1581 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240501 20:09:43.008096 1573 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240501 20:09:43.009344 1573 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 154, Last known committed idx: 153, Time since last communication: 0.000s
I20240501 20:09:43.012936 407 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 138 to 154, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.012429 1414 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.154: config changed from index 138 to 154, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.026157 31905 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 138 to 154, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.026211 31111 catalog_manager.cc:5484] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 138 to 154, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:43.028821 31518 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 138 to 154, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.068042 31387 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 138 to 154, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.138367 31254 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.154: config changed from index 138 to 154, c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 154 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.293706 31646 consensus_queue.cc:237] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.155, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:09:43.319732 407 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240501 20:09:43.320986 31385 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240501 20:09:43.327090 1681 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240501 20:09:43.343142 1638 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240501 20:09:43.377914 31905 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240501 20:09:43.378764 1638 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240501 20:09:43.409682 1581 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.156: config changed from index 154 to 156, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.417833 31909 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 154 to 156, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.421998 31113 catalog_manager.cc:5484] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 154 to 156, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:43.427052 31237 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe38cf1f2ef48e4adca9910e82e2d9e with delete_type TABLET_DATA_TOMBSTONED (TS 341d3bb73a5c4e8f96176a281db4a60f not found in new config with opid_index 156) from {username='slave'} at 127.0.0.1:51884
W20240501 20:09:43.435253 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 95624us
I20240501 20:09:43.435951 31515 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 156. (index mismatch)
I20240501 20:09:43.448210 31387 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 154 to 156, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.464154 1716 tablet_replica.cc:330] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: stopping tablet replica
I20240501 20:09:43.464465 1581 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240501 20:09:43.467257 31514 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 154 to 156, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.468318 1716 raft_consensus.cc:2229] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:43.469051 1716 pending_rounds.cc:62] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: Trying to abort 1 pending ops.
I20240501 20:09:43.469465 1716 pending_rounds.cc:68] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: Aborting op as it isn't in flight: id { term: 1 index: 155 } timestamp: 7022977774370443264 op_type: WRITE_OP write_request { tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\311y{/\277\027\032\221\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a76dc1a32d87911c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022977773804306432 authz_token { token_data: "\010\223\305\312\261\006\"3\n\005slave\022*\n e358c69bd7b74e439c3651a47b25e3e7\020\001\030\001 \001(\001" signature: "<redacted>""\236\227K\265\370\2049\236M\323\253\207\246\236/\337p0\014\356f\035\347\021\375\']\354p\226\215\210^\227\244\342\024\037\020\251\342\275\240\272Wz\265<\341(\007}{\n\006\313\277\246\214\367c\254\321H" signing_key_seq_num: 0 } } request_id { client_id: "c8f385d37c3545fab1a403d8731db195" seq_no: 447 first_incomplete_seq_no: 447 attempt_no: 0 }
I20240501 20:09:43.470504 1716 raft_consensus.cc:2258] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:43.471063 403 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index 154 to 156, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } }
I20240501 20:09:43.480297 1716 ts_tablet_manager.cc:1875] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:43.482357 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfe38cf1f2ef48e4adca9910e82e2d9e with cas_config_opid_index 154: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:43.644600 1716 ts_tablet_manager.cc:1888] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.155
I20240501 20:09:43.644794 1716 log.cc:1196] T dfe38cf1f2ef48e4adca9910e82e2d9e P 341d3bb73a5c4e8f96176a281db4a60f: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/dfe38cf1f2ef48e4adca9910e82e2d9e
I20240501 20:09:43.687386 31104 catalog_manager.cc:4830] TS 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): tablet dfe38cf1f2ef48e4adca9910e82e2d9e (table rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]) successfully deleted
W20240501 20:09:43.798239 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 105014us
W20240501 20:09:43.865306 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 94541us
W20240501 20:09:45.177322 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 46568us
W20240501 20:09:45.460474 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 49678.2us
I20240501 20:09:45.635386 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:45.813954 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:45.877274 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:45.912130 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:45.967725 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:46.012898 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240501 20:09:46.420814 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 50534.8us
W20240501 20:09:46.430213 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 52819.7us
W20240501 20:09:46.512790 31453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 127782us
I20240501 20:09:46.513790 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:47.418053 31904 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 378, Committed index: 378, Last appended: 1.379, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } }
I20240501 20:09:47.454995 403 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 1 index: 380. (index mismatch)
I20240501 20:09:47.455639 1746 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 378, Time since last communication: 0.000s
W20240501 20:09:47.457688 31846 consensus_peers.cc:485] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 -> Peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Couldn't send request to peer db07c3a323eb4a3c919eaf9b19b24986. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7faad10e555240378c4deeb23188128a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:47.459897 31514 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 1 index: 380. (index mismatch)
I20240501 20:09:47.461357 1746 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 378, Time since last communication: 0.000s
I20240501 20:09:47.464509 1605 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.380: config changed from index 164 to 380, NON_VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) added. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } } }
I20240501 20:09:47.473755 403 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.380: config changed from index 164 to 380, NON_VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) added. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } } }
I20240501 20:09:47.473984 31113 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 164 to 380, NON_VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) added. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:47.477504 31774 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 379. Preceding OpId from leader: term: 1 index: 380. (index mismatch)
I20240501 20:09:47.494180 31647 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 378. Preceding OpId from leader: term: 1 index: 380. (index mismatch)
W20240501 20:09:47.510324 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 46112.6us
I20240501 20:09:47.523192 1731 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 378, Time since last communication: 0.000s
I20240501 20:09:47.545377 31518 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.380: config changed from index 164 to 380, NON_VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) added. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } } }
I20240501 20:09:47.551133 31647 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.380: config changed from index 164 to 380, NON_VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) added. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } } }
I20240501 20:09:47.552702 1746 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 380, Last known committed idx: 378, Time since last communication: 0.000s
I20240501 20:09:47.562062 31777 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.380: config changed from index 164 to 380, NON_VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) added. New config: { opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } } }
I20240501 20:09:48.241539 1857 ts_tablet_manager.cc:902] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: Initiating tablet copy from peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
W20240501 20:09:48.271034 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 47951.5us
W20240501 20:09:48.294925 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 69469.6us
I20240501 20:09:48.381950 1857 tablet_copy_client.cc:308] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:37941
I20240501 20:09:48.383183 31930 tablet_copy_service.cc:140] P 582c7a51c0ac43b4934078ebdf434485: Received BeginTabletCopySession request for tablet 7faad10e555240378c4deeb23188128a from peer db07c3a323eb4a3c919eaf9b19b24986 ({username='slave'} at 127.28.136.66:49877)
I20240501 20:09:48.383332 31930 tablet_copy_service.cc:161] P 582c7a51c0ac43b4934078ebdf434485: Beginning new tablet copy session on tablet 7faad10e555240378c4deeb23188128a from peer db07c3a323eb4a3c919eaf9b19b24986 at {username='slave'} at 127.28.136.66:49877: session id = db07c3a323eb4a3c919eaf9b19b24986-7faad10e555240378c4deeb23188128a
I20240501 20:09:48.386085 31930 tablet_copy_source_session.cc:205] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:48.388171 1857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faad10e555240378c4deeb23188128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.397315 1857 tablet_copy_client.cc:783] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:48.397758 1857 tablet_copy_client.cc:647] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: tablet copy: Starting download of 1 WAL segments...
W20240501 20:09:48.436565 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 111475us
I20240501 20:09:48.510418 1857 tablet_copy_client.cc:523] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:48.514603 1857 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap starting.
W20240501 20:09:49.347237 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 74586.4us
W20240501 20:09:50.398345 31846 consensus_peers.cc:485] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 -> Peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Couldn't send request to peer db07c3a323eb4a3c919eaf9b19b24986. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240501 20:09:50.404758 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 70371.3us
W20240501 20:09:50.417084 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 128542us
I20240501 20:09:50.659816 1857 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap replayed 1/1 log segments. Stats: ops{read=391 overwritten=0 applied=390 ignored=0} inserts{seen=1072 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:09:50.668406 1857 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: Bootstrap complete.
I20240501 20:09:50.669808 1857 ts_tablet_manager.cc:1367] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: Time spent bootstrapping tablet: real 2.155s user 0.474s sys 0.031s
I20240501 20:09:50.671205 1857 raft_consensus.cc:348] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } }
I20240501 20:09:50.705518 1857 raft_consensus.cc:729] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Initialized, Role: LEARNER
I20240501 20:09:50.705876 1857 consensus_queue.cc:260] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 390, Last appended: 1.391, Last appended by leader: 391, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 380 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: true } }
I20240501 20:09:50.727025 1857 ts_tablet_manager.cc:1398] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
I20240501 20:09:50.731292 31930 tablet_copy_service.cc:342] P 582c7a51c0ac43b4934078ebdf434485: Request end of tablet copy session db07c3a323eb4a3c919eaf9b19b24986-7faad10e555240378c4deeb23188128a received from {username='slave'} at 127.28.136.66:49877
I20240501 20:09:50.731436 31930 tablet_copy_service.cc:433] P 582c7a51c0ac43b4934078ebdf434485: ending tablet copy session db07c3a323eb4a3c919eaf9b19b24986-7faad10e555240378c4deeb23188128a on tablet 7faad10e555240378c4deeb23188128a with peer db07c3a323eb4a3c919eaf9b19b24986
I20240501 20:09:50.790202 31386 raft_consensus.cc:1203] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.379->[1.380-1.421] Dedup: 1.391->[1.392-1.421]
I20240501 20:09:51.077112 1917 raft_consensus.cc:1050] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: attempting to promote NON_VOTER db07c3a323eb4a3c919eaf9b19b24986 to VOTER
I20240501 20:09:51.078178 1917 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 423, Committed index: 423, Last appended: 1.423, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 424 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } }
I20240501 20:09:51.082310 31774 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 423. Preceding OpId from leader: term: 1 index: 425. (index mismatch)
I20240501 20:09:51.082311 407 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 423. Preceding OpId from leader: term: 1 index: 425. (index mismatch)
I20240501 20:09:51.082903 1922 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 424, Last known committed idx: 423, Time since last communication: 0.000s
I20240501 20:09:51.083372 31515 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 423. Preceding OpId from leader: term: 1 index: 425. (index mismatch)
I20240501 20:09:51.083424 1922 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 424, Last known committed idx: 423, Time since last communication: 0.000s
I20240501 20:09:51.087234 31389 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEARNER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 423. Preceding OpId from leader: term: 1 index: 425. (index mismatch)
I20240501 20:09:51.087929 1917 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 424, Last known committed idx: 423, Time since last communication: 0.000s
I20240501 20:09:51.095086 1731 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.424: config changed from index 380 to 424, db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 424 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.102523 406 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.424: config changed from index 380 to 424, db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 424 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.102746 31774 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.424: config changed from index 380 to 424, db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 424 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.103308 31389 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.424: config changed from index 380 to 424, db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 424 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.103781 31110 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 380 to 424, db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 424 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:51.121970 1731 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 424, Last known committed idx: 423, Time since last communication: 0.000s
W20240501 20:09:51.123559 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 48887.5us
I20240501 20:09:51.125559 31519 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.424: config changed from index 380 to 424, db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 424 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.195506 31647 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 423. Preceding OpId from leader: term: 1 index: 425. (index mismatch)
I20240501 20:09:51.234884 1917 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 424, Last known committed idx: 423, Time since last communication: 0.000s
I20240501 20:09:51.414913 31647 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.424: config changed from index 380 to 424, db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 424 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.624300 31909 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 434, Committed index: 434, Last appended: 1.434, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } }
I20240501 20:09:51.631202 31777 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240501 20:09:51.642956 1970 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240501 20:09:51.649771 403 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240501 20:09:51.650038 31387 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240501 20:09:51.650734 1917 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240501 20:09:51.656069 1917 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240501 20:09:51.657954 1922 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.435: config changed from index 424 to 435, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.661826 403 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 424 to 435, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.672878 31114 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 reported cstate change: config changed from index 424 to 435, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.675819 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7faad10e555240378c4deeb23188128a with cas_config_opid_index 424: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:09:51.679581 31498 tablet_service.cc:1514] Processing DeleteTablet for tablet 7faad10e555240378c4deeb23188128a with delete_type TABLET_DATA_TOMBSTONED (TS ec65c12cd0724cf9aa16fcd61bbedae4 not found in new config with opid_index 435) from {username='slave'} at 127.0.0.1:56620
I20240501 20:09:51.679179 31773 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 424 to 435, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
I20240501 20:09:51.709810 31644 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 434. Preceding OpId from leader: term: 1 index: 435. (index mismatch)
I20240501 20:09:51.721105 1986 tablet_replica.cc:330] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: stopping tablet replica
I20240501 20:09:51.721393 1986 raft_consensus.cc:2229] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:51.722915 1986 raft_consensus.cc:2258] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:51.728781 1986 ts_tablet_manager.cc:1875] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:51.730655 31387 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 424 to 435, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
W20240501 20:09:51.769037 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 54417.5us
I20240501 20:09:51.778064 1970 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 435, Last known committed idx: 434, Time since last communication: 0.000s
I20240501 20:09:51.782729 1986 ts_tablet_manager.cc:1888] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.434
I20240501 20:09:51.785849 1986 log.cc:1196] T 7faad10e555240378c4deeb23188128a P ec65c12cd0724cf9aa16fcd61bbedae4: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/7faad10e555240378c4deeb23188128a
I20240501 20:09:51.789559 31102 catalog_manager.cc:4830] TS ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): tablet 7faad10e555240378c4deeb23188128a (table rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]) successfully deleted
I20240501 20:09:51.816083 31647 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.435: config changed from index 424 to 435, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 435 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } }
W20240501 20:09:52.162611 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 66779.1us
W20240501 20:09:52.391917 31715 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 56592.2us
W20240501 20:09:52.448174 31712 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 107890us
W20240501 20:09:53.070609 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 66798.3us
W20240501 20:09:53.879972 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 45272.3us
W20240501 20:09:53.903577 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 125916us
W20240501 20:09:54.039425 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 53901.2us
I20240501 20:09:54.495066 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:54.511994 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:54.567364 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:54.579838 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:54.708092 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:54.708774 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:54.762077 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:09:55.050558 31715 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 56692.1us
W20240501 20:09:56.063499 31192 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 49902.2us
W20240501 20:09:56.304375 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 75232.3us
W20240501 20:09:56.511018 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 79066.1us
W20240501 20:09:57.736809 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 187441us
W20240501 20:09:57.902662 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 47668.7us
I20240501 20:09:58.046394 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:58.056617 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:58.087283 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:09:58.101158 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:09:58.158071 31192 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 44297us
I20240501 20:09:58.174588 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:58.238595 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:58.278383 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:09:58.304564 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 94708.3us
W20240501 20:09:59.708142 31441 kernel_stack_watchdog.cc:198] Thread 31455 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 208ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f1d70df3980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f1d6da61d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f1d6da827ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f1d6da7e3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f1d6da7c84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f1d791af9e8 kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f1d791b6061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f1d791b5e6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1d746b38cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f1d746b34ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f1d7469f4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
W20240501 20:09:59.817391 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 461946us
W20240501 20:10:00.498952 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 51010.9us
W20240501 20:10:00.660957 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 108413us
W20240501 20:10:01.065069 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 61000.8us
W20240501 20:10:02.316893 31909 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.832s user 0.046s sys 0.034s
W20240501 20:10:02.317135 31909 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.832s user 0.047s sys 0.034s
W20240501 20:10:02.317335 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 145029us
W20240501 20:10:02.365096 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 46979.5us
W20240501 20:10:02.901698 31323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 102041us
W20240501 20:10:02.951467 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 152407us
W20240501 20:10:03.366891 2220 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.003s sys 0.005s
W20240501 20:10:03.367005 2220 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.003s sys 0.005s
W20240501 20:10:04.421304 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 215740us
W20240501 20:10:04.837080 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 53281.2us
W20240501 20:10:05.278728 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 46729.4us
I20240501 20:10:05.711794 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:05.728942 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:05.741514 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:05.742542 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:05.764436 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:10:05.825786 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 62096.9us
I20240501 20:10:05.839488 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:10:05.954483 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 182960us
I20240501 20:10:05.967682 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:10:06.218992 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 100688us
W20240501 20:10:07.023023 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 87632.9us
W20240501 20:10:07.510324 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 55902.9us
W20240501 20:10:08.497761 2414 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.010s sys 0.001s
W20240501 20:10:08.498087 2414 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.601s user 0.010s sys 0.001s
W20240501 20:10:08.593546 31389 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.045s sys 0.083s
W20240501 20:10:08.593752 31389 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.606s user 0.045s sys 0.083s
W20240501 20:10:08.737617 31583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:42361, user_credentials={real_user=slave}} blocked reactor thread for 138531us
W20240501 20:10:08.931034 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 88348.4us
W20240501 20:10:09.384469 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 100423us
W20240501 20:10:09.543274 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 48621.2us
W20240501 20:10:09.917019 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 48426.8us
W20240501 20:10:10.006479 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 85847.6us
W20240501 20:10:10.153756 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 53710.4us
W20240501 20:10:10.168005 31254 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.095s sys 0.147s
W20240501 20:10:10.168203 31254 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s user 0.095s sys 0.147s
W20240501 20:10:10.816794 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:33401, user_credentials={real_user=slave}} blocked reactor thread for 73222.4us
I20240501 20:10:10.965121 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240501 20:10:11.061425 31585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:42527, user_credentials={real_user=slave}} blocked reactor thread for 45449.5us
I20240501 20:10:11.199483 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:11.244751 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:11.269671 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:11.292680 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:11.549041 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:11.629032 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240501 20:10:11.827555 31325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 117288us
W20240501 20:10:11.827910 31322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:44049, user_credentials={real_user=slave}} blocked reactor thread for 95785.2us
W20240501 20:10:12.145743 31452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:37941, user_credentials={real_user=slave}} blocked reactor thread for 53512.7us
W20240501 20:10:12.165411 31455 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:35749, user_credentials={real_user=slave}} blocked reactor thread for 78072.5us
W20240501 20:10:13.191926 31844 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:43505, user_credentials={real_user=slave}} blocked reactor thread for 71034.9us
I20240501 20:10:13.569684 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:13.588830 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:13.611812 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:13.617614 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:13.626891 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:13.633157 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:13.636960 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
18e54ea4106c49c48e2a2a461e4657fd | 127.28.136.126:46583 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.28.136.84:37127 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
341d3bb73a5c4e8f96176a281db4a60f | 127.28.136.65:33401 | HEALTHY | <none> | 2 | 0
582c7a51c0ac43b4934078ebdf434485 | 127.28.136.70:37941 | HEALTHY | <none> | 2 | 0
c42de85026a84ed7b4f462cf25433e01 | 127.28.136.71:35749 | HEALTHY | <none> | 0 | 0
ce9dba79a8c94a7b87bf51e9ed4b5fe1 | 127.28.136.68:44049 | HEALTHY | <none> | 4 | 0
d73d71e1531b4649b363236998019221 | 127.28.136.69:43505 | HEALTHY | <none> | 2 | 0
db07c3a323eb4a3c919eaf9b19b24986 | 127.28.136.66:42527 | HEALTHY | <none> | 3 | 0
ec65c12cd0724cf9aa16fcd61bbedae4 | 127.28.136.67:42361 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.28.136.65 | experimental | 127.28.136.65:33401
local_ip_for_outbound_sockets | 127.28.136.66 | experimental | 127.28.136.66:42527
local_ip_for_outbound_sockets | 127.28.136.67 | experimental | 127.28.136.67:42361
local_ip_for_outbound_sockets | 127.28.136.68 | experimental | 127.28.136.68:44049
local_ip_for_outbound_sockets | 127.28.136.69 | experimental | 127.28.136.69:43505
local_ip_for_outbound_sockets | 127.28.136.70 | experimental | 127.28.136.70:37941
local_ip_for_outbound_sockets | 127.28.136.71 | experimental | 127.28.136.71:35749
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.136.65:33401
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.136.66:42527
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.136.67:42361
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.28.136.68:44049
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.136.69:43505
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.136.70:37941
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.28.136.71:35749
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.28.136.84:37127 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:10:13.748265 29217 log_verifier.cc:126] Checking tablet 2c9d9115e89c437ca7b962201115f087
I20240501 20:10:14.023779 29217 log_verifier.cc:177] Verified matching terms for 272 ops in tablet 2c9d9115e89c437ca7b962201115f087
I20240501 20:10:14.024536 29217 log_verifier.cc:126] Checking tablet 37fbd40e4b9a4d75994b3e138c4506d8
I20240501 20:10:14.539860 29217 log_verifier.cc:177] Verified matching terms for 500 ops in tablet 37fbd40e4b9a4d75994b3e138c4506d8
I20240501 20:10:14.541190 29217 log_verifier.cc:126] Checking tablet 66e635d773d34c9bbf8b2b15c4611b40
I20240501 20:10:14.999045 29217 log_verifier.cc:177] Verified matching terms for 522 ops in tablet 66e635d773d34c9bbf8b2b15c4611b40
I20240501 20:10:15.000211 29217 log_verifier.cc:126] Checking tablet 750c72210b3240839bb156782aa1bdca
I20240501 20:10:15.040256 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 750c72210b3240839bb156782aa1bdca
I20240501 20:10:15.040355 29217 log_verifier.cc:126] Checking tablet 791393d238c3413d9472eecffbf1438a
I20240501 20:10:15.425246 29217 log_verifier.cc:177] Verified matching terms for 389 ops in tablet 791393d238c3413d9472eecffbf1438a
I20240501 20:10:15.426344 29217 log_verifier.cc:126] Checking tablet 7faad10e555240378c4deeb23188128a
I20240501 20:10:16.012234 29217 log_verifier.cc:177] Verified matching terms for 676 ops in tablet 7faad10e555240378c4deeb23188128a
I20240501 20:10:16.013876 29217 log_verifier.cc:126] Checking tablet 87de1a5930394b2ca4ed6e99d01936d7
I20240501 20:10:16.650874 29217 log_verifier.cc:177] Verified matching terms for 692 ops in tablet 87de1a5930394b2ca4ed6e99d01936d7
I20240501 20:10:16.652530 29217 log_verifier.cc:126] Checking tablet 90512cd61a6647c4beb15ffcdd488da3
I20240501 20:10:17.141499 29217 log_verifier.cc:177] Verified matching terms for 546 ops in tablet 90512cd61a6647c4beb15ffcdd488da3
I20240501 20:10:17.142781 29217 log_verifier.cc:126] Checking tablet a3b0a8894c0f46c683461a1857ff27ff
I20240501 20:10:17.552487 29217 log_verifier.cc:177] Verified matching terms for 389 ops in tablet a3b0a8894c0f46c683461a1857ff27ff
I20240501 20:10:17.554064 29217 log_verifier.cc:126] Checking tablet abdd2df68c774be1ab6e39d9d5dbf563
I20240501 20:10:18.099514 29217 log_verifier.cc:177] Verified matching terms for 607 ops in tablet abdd2df68c774be1ab6e39d9d5dbf563
I20240501 20:10:18.100977 29217 log_verifier.cc:126] Checking tablet afb53b647def417c8c9e2378dbeb5dd9
I20240501 20:10:18.688642 29217 log_verifier.cc:177] Verified matching terms for 605 ops in tablet afb53b647def417c8c9e2378dbeb5dd9
I20240501 20:10:18.690373 29217 log_verifier.cc:126] Checking tablet d3b0fefa2f5248f49118f7fb59671b6c
I20240501 20:10:19.234459 29217 log_verifier.cc:177] Verified matching terms for 568 ops in tablet d3b0fefa2f5248f49118f7fb59671b6c
I20240501 20:10:19.236725 29217 log_verifier.cc:126] Checking tablet dda27412e67c4571bd8c5e86a65b787f
I20240501 20:10:19.545233 29217 log_verifier.cc:177] Verified matching terms for 300 ops in tablet dda27412e67c4571bd8c5e86a65b787f
I20240501 20:10:19.546505 29217 log_verifier.cc:126] Checking tablet dfe38cf1f2ef48e4adca9910e82e2d9e
I20240501 20:10:19.864563 29217 log_verifier.cc:177] Verified matching terms for 291 ops in tablet dfe38cf1f2ef48e4adca9910e82e2d9e
I20240501 20:10:19.865285 29217 log_verifier.cc:126] Checking tablet e93ddb68383c43089544a6f0549c83e2
I20240501 20:10:20.213292 29217 log_verifier.cc:177] Verified matching terms for 356 ops in tablet e93ddb68383c43089544a6f0549c83e2
I20240501 20:10:20.214103 29217 log_verifier.cc:126] Checking tablet ef4b7168ca5f4b2ba49ec83e57b48009
I20240501 20:10:20.806118 29217 log_verifier.cc:177] Verified matching terms for 719 ops in tablet ef4b7168ca5f4b2ba49ec83e57b48009
I20240501 20:10:20.820403 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.72:0
--local_ip_for_outbound_sockets=127.28.136.72
--webserver_interface=127.28.136.72
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--builtin_ntp_servers=127.28.136.84:37127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:10:21.197660 2715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:21.197957 2715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:21.198084 2715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:21.206336 2715 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:21.206475 2715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:21.206625 2715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.72
I20240501 20:10:21.215981 2715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:37127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.136.72
--webserver_port=0
--tserver_master_addrs=127.28.136.126:46583
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.72
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:21.217327 2715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:21.219241 2715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:21.233108 2722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:21.233376 2724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:21.234135 2715 server_base.cc:1034] running on GCE node
W20240501 20:10:21.234230 2721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:21.234872 2715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:21.235910 2715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:21.237125 2715 hybrid_clock.cc:648] HybridClock initialized: now 1714594221237061 us; error 57 us; skew 500 ppm
I20240501 20:10:21.237461 2715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:21.239894 2715 webserver.cc:460] Webserver started at http://127.28.136.72:39241/ using document root <none> and password file <none>
I20240501 20:10:21.240665 2715 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:21.240801 2715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:21.241189 2715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:21.243979 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "f27d0fb719e1420c8dbbc10ba0d318f9"
format_stamp: "Formatted at 2024-05-01 20:10:21 on dist-test-slave-sx0n"
I20240501 20:10:21.244866 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "f27d0fb719e1420c8dbbc10ba0d318f9"
format_stamp: "Formatted at 2024-05-01 20:10:21 on dist-test-slave-sx0n"
I20240501 20:10:21.250695 2715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 20:10:21.257074 2730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:21.258612 2715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:21.258802 2715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f27d0fb719e1420c8dbbc10ba0d318f9"
format_stamp: "Formatted at 2024-05-01 20:10:21 on dist-test-slave-sx0n"
I20240501 20:10:21.259027 2715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:21.279091 2715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:21.279999 2715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:21.280297 2715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:21.281553 2715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:21.283120 2715 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:21.283216 2715 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:21.283324 2715 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:21.283391 2715 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:21.329097 2715 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.72:45299
I20240501 20:10:21.329178 2844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.72:45299 every 8 connection(s)
I20240501 20:10:21.330623 2715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:10:21.335067 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 2715
I20240501 20:10:21.335258 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:10:21.346736 2845 heartbeater.cc:347] Connected to a master server at 127.28.136.126:46583
I20240501 20:10:21.347008 2845 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:21.348028 2845 heartbeater.cc:511] Master 127.28.136.126:46583 requested a full tablet report, sending...
I20240501 20:10:21.353252 31109 ts_manager.cc:194] Registered new tserver with Master: f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299)
I20240501 20:10:21.354252 31109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.72:39991
I20240501 20:10:21.905531 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:21.917673 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:21.919888 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:21.945297 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:21.946180 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:21.950199 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:21.951365 2778 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:21.965705 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:22.208110 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:22.209430 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:22.226178 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:22.266940 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:22.267407 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:22.296360 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:22.323189 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:22.334870 2778 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:22.357472 2845 heartbeater.cc:503] Master 127.28.136.126:46583 was elected leader, sending a full tablet report...
I20240501 20:10:22.418134 31644 consensus_queue.cc:237] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 1.291, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:22.421028 31515 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 1 index: 292. (index mismatch)
I20240501 20:10:22.421042 401 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 1 index: 292. (index mismatch)
I20240501 20:10:22.421685 2890 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20240501 20:10:22.423187 2890 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
W20240501 20:10:22.431275 31582 consensus_peers.cc:485] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfe38cf1f2ef48e4adca9910e82e2d9e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.434087 31389 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 1 index: 292. (index mismatch)
I20240501 20:10:22.434161 31910 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 1 index: 292. (index mismatch)
I20240501 20:10:22.434830 2890 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20240501 20:10:22.435708 2890 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20240501 20:10:22.442059 2941 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.292: config changed from index 156 to 292, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.442764 31515 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.292: config changed from index 156 to 292, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.443358 31389 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.292: config changed from index 156 to 292, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.446010 31910 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.292: config changed from index 156 to 292, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.448235 31114 catalog_manager.cc:5484] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 reported cstate change: config changed from index 156 to 292, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.453225 401 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.292: config changed from index 156 to 292, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.470418 31515 consensus_queue.cc:237] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 500, Committed index: 500, Last appended: 1.500, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:22.474169 401 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 500. Preceding OpId from leader: term: 1 index: 501. (index mismatch)
I20240501 20:10:22.474342 31910 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 500. Preceding OpId from leader: term: 1 index: 501. (index mismatch)
I20240501 20:10:22.474328 31389 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 500. Preceding OpId from leader: term: 1 index: 501. (index mismatch)
I20240501 20:10:22.474352 31644 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 500. Preceding OpId from leader: term: 1 index: 501. (index mismatch)
I20240501 20:10:22.474995 2716 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240501 20:10:22.475328 2959 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240501 20:10:22.475957 2959 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240501 20:10:22.477023 2716 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
W20240501 20:10:22.480685 31452 consensus_peers.cc:485] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37fbd40e4b9a4d75994b3e138c4506d8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.482010 31389 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.501: config changed from index 241 to 501, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.481468 2958 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.501: config changed from index 241 to 501, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.491405 31110 catalog_manager.cc:5484] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: config changed from index 241 to 501, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.491420 31910 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.501: config changed from index 241 to 501, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.519629 403 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.501: config changed from index 241 to 501, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.527067 31644 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.501: config changed from index 241 to 501, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.550484 31910 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 676, Committed index: 676, Last appended: 1.676, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:22.554982 31389 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 676. Preceding OpId from leader: term: 1 index: 677. (index mismatch)
I20240501 20:10:22.555115 401 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 676. Preceding OpId from leader: term: 1 index: 677. (index mismatch)
I20240501 20:10:22.555109 31775 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 676. Preceding OpId from leader: term: 1 index: 677. (index mismatch)
I20240501 20:10:22.555635 2966 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 677, Last known committed idx: 676, Time since last communication: 0.000s
I20240501 20:10:22.555965 2853 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 677, Last known committed idx: 676, Time since last communication: 0.000s
I20240501 20:10:22.556427 31644 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 676. Preceding OpId from leader: term: 1 index: 677. (index mismatch)
I20240501 20:10:22.556525 2853 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 677, Last known committed idx: 676, Time since last communication: 0.000s
I20240501 20:10:22.557088 2853 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 677, Last known committed idx: 676, Time since last communication: 0.000s
I20240501 20:10:22.566092 2966 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.677: config changed from index 435 to 677, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.568164 31774 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.677: config changed from index 435 to 677, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.568573 401 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.677: config changed from index 435 to 677, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.574286 31111 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 reported cstate change: config changed from index 435 to 677, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.576686 31644 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.677: config changed from index 435 to 677, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.583724 31389 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.677: config changed from index 435 to 677, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
W20240501 20:10:22.593816 31843 consensus_peers.cc:485] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7faad10e555240378c4deeb23188128a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.600515 31256 consensus_queue.cc:237] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 389, Committed index: 389, Last appended: 1.389, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:22.603989 31644 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 389. Preceding OpId from leader: term: 1 index: 390. (index mismatch)
I20240501 20:10:22.604518 2702 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 390, Last known committed idx: 389, Time since last communication: 0.000s
I20240501 20:10:22.606031 401 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 389. Preceding OpId from leader: term: 1 index: 390. (index mismatch)
I20240501 20:10:22.606456 31515 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 389. Preceding OpId from leader: term: 1 index: 390. (index mismatch)
I20240501 20:10:22.608549 2702 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 390, Last known committed idx: 389, Time since last communication: 0.000s
I20240501 20:10:22.609179 2702 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 390, Last known committed idx: 389, Time since last communication: 0.000s
W20240501 20:10:22.609988 31191 consensus_peers.cc:485] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3b0a8894c0f46c683461a1857ff27ff. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.612263 2702 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.390: config changed from index 120 to 390, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.613128 401 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.390: config changed from index 120 to 390, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.615481 31515 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.390: config changed from index 120 to 390, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.618439 31111 catalog_manager.cc:5484] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 reported cstate change: config changed from index 120 to 390, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.623260 31389 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 389. Preceding OpId from leader: term: 1 index: 390. (index mismatch)
I20240501 20:10:22.624950 31644 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.390: config changed from index 120 to 390, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.625775 2994 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 390, Last known committed idx: 389, Time since last communication: 0.000s
I20240501 20:10:22.639555 31389 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.390: config changed from index 120 to 390, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.648926 31644 consensus_queue.cc:237] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 568, Committed index: 568, Last appended: 1.568, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:22.651567 31515 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 568. Preceding OpId from leader: term: 1 index: 569. (index mismatch)
I20240501 20:10:22.651774 31256 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 568. Preceding OpId from leader: term: 1 index: 569. (index mismatch)
W20240501 20:10:22.652283 31582 consensus_peers.cc:485] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3b0fefa2f5248f49118f7fb59671b6c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.652361 2890 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I20240501 20:10:22.652839 31389 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 568. Preceding OpId from leader: term: 1 index: 569. (index mismatch)
I20240501 20:10:22.652810 2941 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I20240501 20:10:22.653465 2941 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I20240501 20:10:22.658785 2804 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.569: config changed from index -1 to 569, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.659451 31389 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.569: config changed from index -1 to 569, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.661576 31910 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 568. Preceding OpId from leader: term: 1 index: 569. (index mismatch)
I20240501 20:10:22.662160 2890 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 569, Last known committed idx: 568, Time since last communication: 0.000s
I20240501 20:10:22.663893 31515 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.569: config changed from index -1 to 569, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.666594 31111 catalog_manager.cc:5484] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: config changed from index -1 to 569, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.676153 31255 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.569: config changed from index -1 to 569, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.685122 31910 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.569: config changed from index -1 to 569, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:22.906886 3036 ts_tablet_manager.cc:902] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:10:22.908931 3036 tablet_copy_client.cc:308] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:44049
I20240501 20:10:22.914639 31669 tablet_copy_service.cc:140] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Received BeginTabletCopySession request for tablet dfe38cf1f2ef48e4adca9910e82e2d9e from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:53361)
I20240501 20:10:22.914791 31669 tablet_copy_service.cc:161] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Beginning new tablet copy session on tablet dfe38cf1f2ef48e4adca9910e82e2d9e from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:53361: session id = f27d0fb719e1420c8dbbc10ba0d318f9-dfe38cf1f2ef48e4adca9910e82e2d9e
I20240501 20:10:22.916821 31669 tablet_copy_source_session.cc:205] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.921036 3036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe38cf1f2ef48e4adca9910e82e2d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.931540 3036 tablet_copy_client.cc:783] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.932061 3036 tablet_copy_client.cc:647] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:22.935457 3036 tablet_copy_client.cc:523] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:22.939675 3036 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:22.958850 3057 ts_tablet_manager.cc:902] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361)
I20240501 20:10:22.973457 3057 tablet_copy_client.cc:308] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:42361
I20240501 20:10:22.992328 31539 tablet_copy_service.cc:140] P ec65c12cd0724cf9aa16fcd61bbedae4: Received BeginTabletCopySession request for tablet 37fbd40e4b9a4d75994b3e138c4506d8 from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:48059)
I20240501 20:10:22.992516 31539 tablet_copy_service.cc:161] P ec65c12cd0724cf9aa16fcd61bbedae4: Beginning new tablet copy session on tablet 37fbd40e4b9a4d75994b3e138c4506d8 from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:48059: session id = f27d0fb719e1420c8dbbc10ba0d318f9-37fbd40e4b9a4d75994b3e138c4506d8
I20240501 20:10:22.995553 31539 tablet_copy_source_session.cc:205] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.998710 3036 log.cc:824] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:23.009781 3057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fbd40e4b9a4d75994b3e138c4506d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.014636 3057 tablet_copy_client.cc:783] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.014912 3057 tablet_copy_client.cc:647] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.017788 3057 tablet_copy_client.cc:523] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.021530 3057 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:23.097133 3076 ts_tablet_manager.cc:902] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:10:23.097169 3074 ts_tablet_manager.cc:902] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401)
I20240501 20:10:23.139447 3076 tablet_copy_client.cc:308] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:37941
I20240501 20:10:23.141331 3074 tablet_copy_client.cc:308] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:33401
I20240501 20:10:23.148166 31930 tablet_copy_service.cc:140] P 582c7a51c0ac43b4934078ebdf434485: Received BeginTabletCopySession request for tablet 7faad10e555240378c4deeb23188128a from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:34565)
I20240501 20:10:23.148339 31930 tablet_copy_service.cc:161] P 582c7a51c0ac43b4934078ebdf434485: Beginning new tablet copy session on tablet 7faad10e555240378c4deeb23188128a from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:34565: session id = f27d0fb719e1420c8dbbc10ba0d318f9-7faad10e555240378c4deeb23188128a
I20240501 20:10:23.151211 31930 tablet_copy_source_session.cc:205] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:23.159914 3076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7faad10e555240378c4deeb23188128a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.160328 31278 tablet_copy_service.cc:140] P 341d3bb73a5c4e8f96176a281db4a60f: Received BeginTabletCopySession request for tablet a3b0a8894c0f46c683461a1857ff27ff from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:59809)
I20240501 20:10:23.160504 31278 tablet_copy_service.cc:161] P 341d3bb73a5c4e8f96176a281db4a60f: Beginning new tablet copy session on tablet a3b0a8894c0f46c683461a1857ff27ff from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:59809: session id = f27d0fb719e1420c8dbbc10ba0d318f9-a3b0a8894c0f46c683461a1857ff27ff
I20240501 20:10:23.163537 31278 tablet_copy_source_session.cc:205] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:23.165189 3074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3b0a8894c0f46c683461a1857ff27ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.170774 3074 tablet_copy_client.cc:783] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.171080 3074 tablet_copy_client.cc:647] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.174363 3074 tablet_copy_client.cc:523] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.179857 3074 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:23.182484 3076 tablet_copy_client.cc:783] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.183902 3076 tablet_copy_client.cc:647] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.190193 3076 tablet_copy_client.cc:523] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.196990 3076 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:23.226789 3087 ts_tablet_manager.cc:902] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:10:23.286437 3087 tablet_copy_client.cc:308] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:44049
I20240501 20:10:23.287354 31669 tablet_copy_service.cc:140] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Received BeginTabletCopySession request for tablet d3b0fefa2f5248f49118f7fb59671b6c from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:53361)
I20240501 20:10:23.287550 31669 tablet_copy_service.cc:161] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Beginning new tablet copy session on tablet d3b0fefa2f5248f49118f7fb59671b6c from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:53361: session id = f27d0fb719e1420c8dbbc10ba0d318f9-d3b0fefa2f5248f49118f7fb59671b6c
I20240501 20:10:23.291081 31669 tablet_copy_source_session.cc:205] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:23.294574 3087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b0fefa2f5248f49118f7fb59671b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.365535 3087 tablet_copy_client.cc:783] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.365986 3087 tablet_copy_client.cc:647] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.377591 3087 tablet_copy_client.cc:523] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.387326 3087 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:23.543345 3036 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=292 overwritten=0 applied=292 ignored=0} inserts{seen=974 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:23.545658 3036 tablet_bootstrap.cc:492] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:23.548461 3036 ts_tablet_manager.cc:1367] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.609s user 0.367s sys 0.024s
I20240501 20:10:23.575819 3036 raft_consensus.cc:348] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:23.580154 3036 raft_consensus.cc:729] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:23.581020 3036 consensus_queue.cc:260] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 292, Last appended: 1.292, Last appended by leader: 292, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 292 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:23.599562 3036 ts_tablet_manager.cc:1398] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.051s user 0.023s sys 0.000s
I20240501 20:10:23.600805 31669 tablet_copy_service.cc:342] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-dfe38cf1f2ef48e4adca9910e82e2d9e received from {username='slave'} at 127.28.136.72:53361
I20240501 20:10:23.600958 31669 tablet_copy_service.cc:433] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-dfe38cf1f2ef48e4adca9910e82e2d9e on tablet dfe38cf1f2ef48e4adca9910e82e2d9e with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:23.747184 2798 raft_consensus.cc:1203] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.291->[1.292-1.292] Dedup: 1.292->[]
I20240501 20:10:23.905153 3057 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=501 overwritten=0 applied=501 ignored=0} inserts{seen=1183 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:23.905812 3057 tablet_bootstrap.cc:492] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:23.906258 3057 ts_tablet_manager.cc:1367] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.885s user 0.570s sys 0.106s
I20240501 20:10:23.907545 3057 raft_consensus.cc:348] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:23.907971 3057 raft_consensus.cc:729] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:23.908264 3057 consensus_queue.cc:260] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 501, Last appended: 1.501, Last appended by leader: 501, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:23.909032 3057 ts_tablet_manager.cc:1398] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:23.909827 31539 tablet_copy_service.cc:342] P ec65c12cd0724cf9aa16fcd61bbedae4: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-37fbd40e4b9a4d75994b3e138c4506d8 received from {username='slave'} at 127.28.136.72:48059
I20240501 20:10:23.909961 31539 tablet_copy_service.cc:433] P ec65c12cd0724cf9aa16fcd61bbedae4: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-37fbd40e4b9a4d75994b3e138c4506d8 on tablet 37fbd40e4b9a4d75994b3e138c4506d8 with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:23.995839 2798 raft_consensus.cc:1203] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.500->[1.501-1.501] Dedup: 1.501->[]
I20240501 20:10:24.060163 3074 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=390 overwritten=0 applied=390 ignored=0} inserts{seen=1059 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:24.061745 3074 tablet_bootstrap.cc:492] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:24.062170 3074 ts_tablet_manager.cc:1367] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.882s user 0.458s sys 0.060s
I20240501 20:10:24.063141 3074 raft_consensus.cc:348] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.063580 3074 raft_consensus.cc:729] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:24.063845 3074 consensus_queue.cc:260] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 390, Last appended: 1.390, Last appended by leader: 390, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 390 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.067642 3074 ts_tablet_manager.cc:1398] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:24.070814 31278 tablet_copy_service.cc:342] P 341d3bb73a5c4e8f96176a281db4a60f: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-a3b0a8894c0f46c683461a1857ff27ff received from {username='slave'} at 127.28.136.72:59809
I20240501 20:10:24.070956 31278 tablet_copy_service.cc:433] P 341d3bb73a5c4e8f96176a281db4a60f: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-a3b0a8894c0f46c683461a1857ff27ff on tablet a3b0a8894c0f46c683461a1857ff27ff with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:24.096549 2798 raft_consensus.cc:1203] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.389->[1.390-1.390] Dedup: 1.390->[]
I20240501 20:10:24.172623 2804 raft_consensus.cc:1050] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:24.173846 2804 consensus_queue.cc:237] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.292, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:24.194396 31515 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240501 20:10:24.195319 3004 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240501 20:10:24.205711 31389 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240501 20:10:24.206306 401 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240501 20:10:24.206656 3158 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240501 20:10:24.207865 31910 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240501 20:10:24.208448 2798 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch)
I20240501 20:10:24.212205 3004 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240501 20:10:24.213524 3004 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240501 20:10:24.214395 3004 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s
I20240501 20:10:24.227782 2804 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.293: config changed from index 292 to 293, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.235117 31910 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.241861 31111 catalog_manager.cc:5484] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 292 to 293, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.255013 2798 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.261701 3087 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=569 overwritten=0 applied=569 ignored=0} inserts{seen=1254 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:24.262307 3087 tablet_bootstrap.cc:492] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:24.262765 3087 ts_tablet_manager.cc:1367] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.876s user 0.631s sys 0.080s
I20240501 20:10:24.263690 3087 raft_consensus.cc:348] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.264159 3087 raft_consensus.cc:729] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:24.264444 3087 consensus_queue.cc:260] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 569, Last appended: 1.569, Last appended by leader: 569, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 569 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.266772 3087 ts_tablet_manager.cc:1398] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:10:24.267503 31669 tablet_copy_service.cc:342] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-d3b0fefa2f5248f49118f7fb59671b6c received from {username='slave'} at 127.28.136.72:53361
I20240501 20:10:24.267639 31669 tablet_copy_service.cc:433] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-d3b0fefa2f5248f49118f7fb59671b6c on tablet d3b0fefa2f5248f49118f7fb59671b6c with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:24.292649 31515 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.297228 31389 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.306109 403 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index 292 to 293, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } attrs { replace: true } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.343957 31644 consensus_queue.cc:237] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 293, Committed index: 293, Last appended: 1.293, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:24.346872 31515 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:10:24.348156 403 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:10:24.348143 31389 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:10:24.348889 2804 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:10:24.349833 2798 raft_consensus.cc:1261] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 293. Preceding OpId from leader: term: 1 index: 294. (index mismatch)
I20240501 20:10:24.350246 3098 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:10:24.350993 3166 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:10:24.355623 3166 consensus_queue.cc:1035] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 294, Last known committed idx: 293, Time since last communication: 0.000s
I20240501 20:10:24.355374 2804 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.294: config changed from index 293 to 294, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.357282 2798 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 293 to 294, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.359573 31389 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 293 to 294, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.363698 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfe38cf1f2ef48e4adca9910e82e2d9e with cas_config_opid_index 293: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:24.364111 31515 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 293 to 294, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.365237 403 raft_consensus.cc:2932] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.294: config changed from index 293 to 294, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New config: { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.367779 31115 catalog_manager.cc:5484] T dfe38cf1f2ef48e4adca9910e82e2d9e P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: config changed from index 293 to 294, VOTER 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70) evicted. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.373026 31888 tablet_service.cc:1514] Processing DeleteTablet for tablet dfe38cf1f2ef48e4adca9910e82e2d9e with delete_type TABLET_DATA_TOMBSTONED (TS 582c7a51c0ac43b4934078ebdf434485 not found in new config with opid_index 294) from {username='slave'} at 127.0.0.1:46678
I20240501 20:10:24.381032 3176 tablet_replica.cc:330] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: stopping tablet replica
I20240501 20:10:24.381369 3176 raft_consensus.cc:2229] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:24.381687 3176 raft_consensus.cc:2258] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.384902 3176 ts_tablet_manager.cc:1875] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:24.391996 3176 ts_tablet_manager.cc:1888] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.293
I20240501 20:10:24.392254 3176 log.cc:1196] T dfe38cf1f2ef48e4adca9910e82e2d9e P 582c7a51c0ac43b4934078ebdf434485: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/wals/dfe38cf1f2ef48e4adca9910e82e2d9e
I20240501 20:10:24.393307 31101 catalog_manager.cc:4830] TS 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): tablet dfe38cf1f2ef48e4adca9910e82e2d9e (table rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]) successfully deleted
I20240501 20:10:24.433387 3076 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=677 overwritten=0 applied=677 ignored=0} inserts{seen=1356 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:24.433879 3076 tablet_bootstrap.cc:492] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:24.434263 3076 ts_tablet_manager.cc:1367] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 1.237s user 0.750s sys 0.071s
I20240501 20:10:24.435034 3076 raft_consensus.cc:348] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.435353 3076 raft_consensus.cc:729] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:24.435619 3076 consensus_queue.cc:260] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 677, Last appended: 1.677, Last appended by leader: 677, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.436125 3076 ts_tablet_manager.cc:1398] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:24.436882 31930 tablet_copy_service.cc:342] P 582c7a51c0ac43b4934078ebdf434485: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-7faad10e555240378c4deeb23188128a received from {username='slave'} at 127.28.136.72:34565
I20240501 20:10:24.437032 31930 tablet_copy_service.cc:433] P 582c7a51c0ac43b4934078ebdf434485: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-7faad10e555240378c4deeb23188128a on tablet 7faad10e555240378c4deeb23188128a with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:24.454368 31775 consensus_queue.cc:237] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.457197 31255 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:24.457782 3209 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:24.458083 31644 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:24.458709 3209 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:24.472460 2980 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.473181 31255 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
W20240501 20:10:24.475881 31712 consensus_peers.cc:485] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 750c72210b3240839bb156782aa1bdca. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:24.477114 31389 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:24.477435 403 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:24.477766 3209 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:24.478523 31644 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.482064 31110 catalog_manager.cc:5484] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index -1 to 2, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "d73d71e1531b4649b363236998019221" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.482363 2980 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:24.497236 403 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.499038 31389 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.507445 3146 raft_consensus.cc:1050] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:24.508442 3146 consensus_queue.cc:237] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 501, Committed index: 501, Last appended: 1.501, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:24.513180 2798 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240501 20:10:24.513701 3147 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
I20240501 20:10:24.514890 31644 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240501 20:10:24.515399 3147 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
I20240501 20:10:24.519361 403 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240501 20:10:24.519955 31910 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240501 20:10:24.519939 3147 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
I20240501 20:10:24.521694 3147 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
I20240501 20:10:24.526860 3147 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.502: config changed from index 501 to 502, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.527355 31910 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index 501 to 502, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.530215 403 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index 501 to 502, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.534641 31115 catalog_manager.cc:5484] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 501 to 502, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.537245 31389 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240501 20:10:24.537827 2798 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index 501 to 502, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.538704 31644 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index 501 to 502, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.542310 3146 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
I20240501 20:10:24.548965 31389 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index 501 to 502, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.552294 2798 raft_consensus.cc:1203] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.568->[1.569-1.569] Dedup: 1.569->[]
I20240501 20:10:24.560027 31515 consensus_queue.cc:237] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 502, Committed index: 502, Last appended: 1.502, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:24.562309 2798 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240501 20:10:24.562752 403 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240501 20:10:24.563374 31910 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240501 20:10:24.563560 3146 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240501 20:10:24.563908 3234 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240501 20:10:24.564074 31644 raft_consensus.cc:1261] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer ec65c12cd0724cf9aa16fcd61bbedae4: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240501 20:10:24.565943 3146 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240501 20:10:24.567628 31910 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.566761 3231 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 LEADER]: Committing config change with OpId 1.503: config changed from index 502 to 503, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.570796 3146 consensus_queue.cc:1035] T 37fbd40e4b9a4d75994b3e138c4506d8 P ec65c12cd0724cf9aa16fcd61bbedae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240501 20:10:24.572502 31644 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.573531 31114 catalog_manager.cc:5484] T 37fbd40e4b9a4d75994b3e138c4506d8 P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 502 to 503, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" committed_config { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.573913 31102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37fbd40e4b9a4d75994b3e138c4506d8 with cas_config_opid_index 502: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:24.576898 403 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.578132 2798 raft_consensus.cc:2932] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.579633 31367 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fbd40e4b9a4d75994b3e138c4506d8 with delete_type TABLET_DATA_TOMBSTONED (TS db07c3a323eb4a3c919eaf9b19b24986 not found in new config with opid_index 503) from {username='slave'} at 127.0.0.1:51676
I20240501 20:10:24.580356 3243 tablet_replica.cc:330] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: stopping tablet replica
I20240501 20:10:24.580642 3243 raft_consensus.cc:2229] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:24.580986 3243 raft_consensus.cc:2258] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.587404 3243 ts_tablet_manager.cc:1875] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:24.592798 3243 ts_tablet_manager.cc:1888] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.502
I20240501 20:10:24.592933 3243 log.cc:1196] T 37fbd40e4b9a4d75994b3e138c4506d8 P db07c3a323eb4a3c919eaf9b19b24986: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/37fbd40e4b9a4d75994b3e138c4506d8
I20240501 20:10:24.593715 31104 catalog_manager.cc:4830] TS db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): tablet 37fbd40e4b9a4d75994b3e138c4506d8 (table rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]) successfully deleted
I20240501 20:10:24.618412 31644 consensus_queue.cc:237] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 1.356, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.621261 403 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
I20240501 20:10:24.621510 31775 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
I20240501 20:10:24.621618 31910 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
W20240501 20:10:24.621923 31582 consensus_peers.cc:485] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e93ddb68383c43089544a6f0549c83e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:24.622043 3098 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20240501 20:10:24.622206 31389 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
I20240501 20:10:24.622829 3098 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20240501 20:10:24.623114 3158 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20240501 20:10:24.623381 2804 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20240501 20:10:24.626060 2804 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.357: config changed from index 98 to 357, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.627516 31910 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 98 to 357, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.629631 403 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 98 to 357, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.630376 31389 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 98 to 357, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.634008 31114 catalog_manager.cc:5484] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 98 to 357, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:24.637822 31775 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 98 to 357, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:24.668715 3283 raft_consensus.cc:1050] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:24.669863 3283 consensus_queue.cc:237] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 390, Committed index: 390, Last appended: 1.390, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:24.673424 2798 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20240501 20:10:24.673478 403 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20240501 20:10:24.673491 31515 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20240501 20:10:24.673843 31389 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20240501 20:10:24.674074 3283 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20240501 20:10:24.674408 3126 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20240501 20:10:24.674736 31646 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 390. Preceding OpId from leader: term: 1 index: 391. (index mismatch)
I20240501 20:10:24.674952 3126 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20240501 20:10:24.676076 3282 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20240501 20:10:24.680003 3287 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391, Last known committed idx: 390, Time since last communication: 0.000s
I20240501 20:10:24.682547 3282 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.391: config changed from index 390 to 391, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.683001 31389 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 390 to 391, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.687027 403 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 390 to 391, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.687165 31515 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 390 to 391, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.688221 31115 catalog_manager.cc:5484] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index 390 to 391, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:24.707670 31646 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 390 to 391, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.713171 2798 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.391: config changed from index 390 to 391, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 391 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.714582 31255 consensus_queue.cc:237] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 391, Committed index: 391, Last appended: 1.391, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:24.716652 403 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20240501 20:10:24.716719 31515 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20240501 20:10:24.716902 31389 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20240501 20:10:24.717555 3287 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20240501 20:10:24.717923 3282 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20240501 20:10:24.718873 3126 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20240501 20:10:24.721772 2798 raft_consensus.cc:1261] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 391. Preceding OpId from leader: term: 1 index: 392. (index mismatch)
I20240501 20:10:24.722334 3287 consensus_queue.cc:1035] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 392, Last known committed idx: 391, Time since last communication: 0.000s
I20240501 20:10:24.723222 3283 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P 341d3bb73a5c4e8f96176a281db4a60f [term 1 LEADER]: Committing config change with OpId 1.392: config changed from index 391 to 392, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.724160 2798 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.727412 31104 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3b0a8894c0f46c683461a1857ff27ff with cas_config_opid_index 391: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:24.728096 31110 catalog_manager.cc:5484] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 reported cstate change: config changed from index 391 to 392, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.728896 31515 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.733459 31389 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.740518 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b0a8894c0f46c683461a1857ff27ff with delete_type TABLET_DATA_TOMBSTONED (TS ce9dba79a8c94a7b87bf51e9ed4b5fe1 not found in new config with opid_index 392) from {username='slave'} at 127.0.0.1:44544
I20240501 20:10:24.740401 403 raft_consensus.cc:2932] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.392: config changed from index 391 to 392, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:24.759264 3300 tablet_replica.cc:330] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: stopping tablet replica
I20240501 20:10:24.759510 3300 raft_consensus.cc:2229] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:24.759810 3300 raft_consensus.cc:2258] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.761859 3300 ts_tablet_manager.cc:1875] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:24.767146 3300 ts_tablet_manager.cc:1888] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.391
I20240501 20:10:24.767303 3300 log.cc:1196] T a3b0a8894c0f46c683461a1857ff27ff P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/a3b0a8894c0f46c683461a1857ff27ff
I20240501 20:10:24.768090 31101 catalog_manager.cc:4830] TS ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): tablet a3b0a8894c0f46c683461a1857ff27ff (table rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]) successfully deleted
I20240501 20:10:24.919656 2798 raft_consensus.cc:1203] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.676->[1.677-1.677] Dedup: 1.677->[]
I20240501 20:10:24.950634 3302 ts_tablet_manager.cc:902] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:10:24.952872 3302 tablet_copy_client.cc:308] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:43505
I20240501 20:10:24.958315 31800 tablet_copy_service.cc:140] P d73d71e1531b4649b363236998019221: Received BeginTabletCopySession request for tablet 750c72210b3240839bb156782aa1bdca from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:53079)
I20240501 20:10:24.958499 31800 tablet_copy_service.cc:161] P d73d71e1531b4649b363236998019221: Beginning new tablet copy session on tablet 750c72210b3240839bb156782aa1bdca from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:53079: session id = f27d0fb719e1420c8dbbc10ba0d318f9-750c72210b3240839bb156782aa1bdca
I20240501 20:10:24.961005 31800 tablet_copy_source_session.cc:205] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:24.962807 3302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750c72210b3240839bb156782aa1bdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.967962 3302 tablet_copy_client.cc:783] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:24.968302 3302 tablet_copy_client.cc:647] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:24.970505 3302 tablet_copy_client.cc:523] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:24.973595 3302 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:24.978947 3302 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:24.979377 3302 tablet_bootstrap.cc:492] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:24.979607 3302 ts_tablet_manager.cc:1367] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:10:24.980336 3302 raft_consensus.cc:348] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.980628 3302 raft_consensus.cc:729] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:24.980868 3302 consensus_queue.cc:260] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:24.981966 3302 ts_tablet_manager.cc:1398] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:10:24.982828 31800 tablet_copy_service.cc:342] P d73d71e1531b4649b363236998019221: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-750c72210b3240839bb156782aa1bdca received from {username='slave'} at 127.28.136.72:53079
I20240501 20:10:24.983001 31800 tablet_copy_service.cc:433] P d73d71e1531b4649b363236998019221: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-750c72210b3240839bb156782aa1bdca on tablet 750c72210b3240839bb156782aa1bdca with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:25.010854 3302 ts_tablet_manager.cc:902] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:10:25.027127 3302 tablet_copy_client.cc:308] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:44049
I20240501 20:10:25.027879 31669 tablet_copy_service.cc:140] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Received BeginTabletCopySession request for tablet e93ddb68383c43089544a6f0549c83e2 from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:53361)
I20240501 20:10:25.028025 31669 tablet_copy_service.cc:161] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Beginning new tablet copy session on tablet e93ddb68383c43089544a6f0549c83e2 from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:53361: session id = f27d0fb719e1420c8dbbc10ba0d318f9-e93ddb68383c43089544a6f0549c83e2
I20240501 20:10:25.030093 31669 tablet_copy_source_session.cc:205] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:25.031360 3302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93ddb68383c43089544a6f0549c83e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.034119 31775 consensus_queue.cc:237] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 692, Committed index: 692, Last appended: 1.692, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:25.036170 3302 tablet_copy_client.cc:783] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:25.036435 3302 tablet_copy_client.cc:647] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:25.039971 31515 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 692. Preceding OpId from leader: term: 1 index: 693. (index mismatch)
I20240501 20:10:25.040694 31646 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 692. Preceding OpId from leader: term: 1 index: 693. (index mismatch)
I20240501 20:10:25.040916 2980 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 693, Last known committed idx: 692, Time since last communication: 0.000s
W20240501 20:10:25.041200 31712 consensus_peers.cc:485] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87de1a5930394b2ca4ed6e99d01936d7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:25.041541 3302 tablet_copy_client.cc:523] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:25.042064 31255 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 692. Preceding OpId from leader: term: 1 index: 693. (index mismatch)
I20240501 20:10:25.042793 3209 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 693, Last known committed idx: 692, Time since last communication: 0.000s
I20240501 20:10:25.044224 2980 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 693, Last known committed idx: 692, Time since last communication: 0.000s
I20240501 20:10:25.046047 3302 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:25.047168 31389 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 692. Preceding OpId from leader: term: 1 index: 693. (index mismatch)
I20240501 20:10:25.047899 2980 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 693, Last known committed idx: 692, Time since last communication: 0.000s
I20240501 20:10:25.054316 2980 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 1 LEADER]: Committing config change with OpId 1.693: config changed from index -1 to 693, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.055670 31255 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.693: config changed from index -1 to 693, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.066378 31389 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.693: config changed from index -1 to 693, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.077230 31515 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.693: config changed from index -1 to 693, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.079030 31110 catalog_manager.cc:5484] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index -1 to 693, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "d73d71e1531b4649b363236998019221" committed_config { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.089046 31646 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.693: config changed from index -1 to 693, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.098670 3166 raft_consensus.cc:1050] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:25.099614 3166 consensus_queue.cc:237] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 569, Committed index: 569, Last appended: 1.569, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 570 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:25.118140 2798 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 569. Preceding OpId from leader: term: 1 index: 570. (index mismatch)
I20240501 20:10:25.118592 31255 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 569. Preceding OpId from leader: term: 1 index: 570. (index mismatch)
I20240501 20:10:25.118993 31515 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 569. Preceding OpId from leader: term: 1 index: 570. (index mismatch)
I20240501 20:10:25.119093 2804 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 570, Last known committed idx: 569, Time since last communication: 0.000s
I20240501 20:10:25.119910 2804 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 570, Last known committed idx: 569, Time since last communication: 0.000s
I20240501 20:10:25.120939 3158 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 570, Last known committed idx: 569, Time since last communication: 0.000s
I20240501 20:10:25.127398 31389 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 569. Preceding OpId from leader: term: 1 index: 570. (index mismatch)
I20240501 20:10:25.128121 3158 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 570, Last known committed idx: 569, Time since last communication: 0.000s
I20240501 20:10:25.132591 31910 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 569. Preceding OpId from leader: term: 1 index: 570. (index mismatch)
I20240501 20:10:25.133621 3158 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 570, Last known committed idx: 569, Time since last communication: 0.000s
I20240501 20:10:25.143846 2804 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.570: config changed from index 569 to 570, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 570 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.144604 31255 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.570: config changed from index 569 to 570, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 570 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.149888 31514 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.570: config changed from index 569 to 570, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 570 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.150141 31910 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.570: config changed from index 569 to 570, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 570 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.153795 31110 catalog_manager.cc:5484] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 569 to 570, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 570 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:25.157954 2798 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.570: config changed from index 569 to 570, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 570 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.161247 31389 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.570: config changed from index 569 to 570, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 570 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.172706 31646 consensus_queue.cc:237] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 570, Committed index: 570, Last appended: 1.570, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:25.175900 2798 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 570. Preceding OpId from leader: term: 1 index: 571. (index mismatch)
I20240501 20:10:25.176468 31255 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 570. Preceding OpId from leader: term: 1 index: 571. (index mismatch)
I20240501 20:10:25.176602 31389 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 570. Preceding OpId from leader: term: 1 index: 571. (index mismatch)
I20240501 20:10:25.176865 3098 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I20240501 20:10:25.177687 3098 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I20240501 20:10:25.178825 31910 raft_consensus.cc:1261] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 570. Preceding OpId from leader: term: 1 index: 571. (index mismatch)
I20240501 20:10:25.182189 3158 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I20240501 20:10:25.185262 3166 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.571: config changed from index 570 to 571, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.185990 31256 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.571: config changed from index 570 to 571, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.189046 31389 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.571: config changed from index 570 to 571, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.190304 2798 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.571: config changed from index 570 to 571, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.192836 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3b0fefa2f5248f49118f7fb59671b6c with cas_config_opid_index 570: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:25.195148 3158 consensus_queue.cc:1035] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 571, Last known committed idx: 570, Time since last communication: 0.000s
I20240501 20:10:25.195881 31110 catalog_manager.cc:5484] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 570 to 571, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:25.197047 31909 raft_consensus.cc:2932] T d3b0fefa2f5248f49118f7fb59671b6c P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.571: config changed from index 570 to 571, VOTER ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67) evicted. New config: { opid_index: 571 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.200359 31498 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b0fefa2f5248f49118f7fb59671b6c with delete_type TABLET_DATA_TOMBSTONED (TS ec65c12cd0724cf9aa16fcd61bbedae4 not found in new config with opid_index 571) from {username='slave'} at 127.0.0.1:56620
I20240501 20:10:25.218214 3361 tablet_replica.cc:330] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: stopping tablet replica
I20240501 20:10:25.218550 3361 raft_consensus.cc:2229] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.218931 3361 raft_consensus.cc:2258] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.222016 3361 ts_tablet_manager.cc:1875] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.235049 3361 ts_tablet_manager.cc:1888] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.570
I20240501 20:10:25.238576 3361 log.cc:1196] T d3b0fefa2f5248f49118f7fb59671b6c P ec65c12cd0724cf9aa16fcd61bbedae4: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/d3b0fefa2f5248f49118f7fb59671b6c
I20240501 20:10:25.239878 31102 catalog_manager.cc:4830] TS ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): tablet d3b0fefa2f5248f49118f7fb59671b6c (table rebalance_test_table_0 [id=c1738f5c265547cab42f97225ef6761e]) successfully deleted
I20240501 20:10:25.312891 31644 consensus_queue.cc:237] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 1.300, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:25.319602 31775 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch)
W20240501 20:10:25.320245 31582 consensus_peers.cc:485] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dda27412e67c4571bd8c5e86a65b787f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:25.320547 403 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch)
I20240501 20:10:25.320554 3166 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20240501 20:10:25.321388 3166 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20240501 20:10:25.321832 31910 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch)
I20240501 20:10:25.322803 3166 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20240501 20:10:25.325578 31256 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch)
I20240501 20:10:25.326428 3166 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s
I20240501 20:10:25.337240 3166 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.301: config changed from index 118 to 301, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.339257 31774 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 118 to 301, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.341976 401 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 118 to 301, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.344238 31256 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 118 to 301, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.349280 31115 catalog_manager.cc:5484] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 reported cstate change: config changed from index 118 to 301, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.368881 31910 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 118 to 301, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:25.392961 3398 raft_consensus.cc:1050] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:25.394037 3398 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 677, Committed index: 677, Last appended: 1.677, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:25.397714 2798 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 677. Preceding OpId from leader: term: 1 index: 678. (index mismatch)
I20240501 20:10:25.399849 31644 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 677. Preceding OpId from leader: term: 1 index: 678. (index mismatch)
I20240501 20:10:25.400997 3399 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 678, Last known committed idx: 677, Time since last communication: 0.000s
I20240501 20:10:25.401705 31389 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 677. Preceding OpId from leader: term: 1 index: 678. (index mismatch)
I20240501 20:10:25.402122 3399 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 678, Last known committed idx: 677, Time since last communication: 0.000s
I20240501 20:10:25.405292 3399 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 678, Last known committed idx: 677, Time since last communication: 0.000s
I20240501 20:10:25.414608 401 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 677. Preceding OpId from leader: term: 1 index: 678. (index mismatch)
I20240501 20:10:25.415719 31775 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 677. Preceding OpId from leader: term: 1 index: 678. (index mismatch)
I20240501 20:10:25.416698 3401 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 678, Last known committed idx: 677, Time since last communication: 0.000s
I20240501 20:10:25.418972 3397 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 678, Last known committed idx: 677, Time since last communication: 0.000s
I20240501 20:10:25.424005 3397 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.678: config changed from index 677 to 678, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.430522 2798 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.678: config changed from index 677 to 678, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.431064 31644 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.678: config changed from index 677 to 678, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.432345 31774 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.678: config changed from index 677 to 678, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.438472 31110 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 reported cstate change: config changed from index 677 to 678, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.439018 31389 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.678: config changed from index 677 to 678, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.442283 401 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.678: config changed from index 677 to 678, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.451668 31910 consensus_queue.cc:237] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 678, Committed index: 678, Last appended: 1.678, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 679 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:25.455922 2798 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 678. Preceding OpId from leader: term: 1 index: 679. (index mismatch)
I20240501 20:10:25.455962 31644 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 678. Preceding OpId from leader: term: 1 index: 679. (index mismatch)
I20240501 20:10:25.455390 401 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 678. Preceding OpId from leader: term: 1 index: 679. (index mismatch)
I20240501 20:10:25.457159 3397 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 679, Last known committed idx: 678, Time since last communication: 0.000s
I20240501 20:10:25.458910 3397 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 679, Last known committed idx: 678, Time since last communication: 0.000s
I20240501 20:10:25.459376 3397 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 679, Last known committed idx: 678, Time since last communication: 0.000s
I20240501 20:10:25.462394 403 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.679: config changed from index 678 to 679, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 679 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.462563 2798 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.679: config changed from index 678 to 679, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 679 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.467622 3397 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [term 1 LEADER]: Committing config change with OpId 1.679: config changed from index 678 to 679, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 679 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.468899 31114 catalog_manager.cc:5484] T 7faad10e555240378c4deeb23188128a P f27d0fb719e1420c8dbbc10ba0d318f9 reported cstate change: config changed from index 678 to 679, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New cstate: current_term: 1 leader_uuid: "582c7a51c0ac43b4934078ebdf434485" committed_config { opid_index: 679 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.472654 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7faad10e555240378c4deeb23188128a with cas_config_opid_index 678: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:25.474062 31759 tablet_service.cc:1514] Processing DeleteTablet for tablet 7faad10e555240378c4deeb23188128a with delete_type TABLET_DATA_TOMBSTONED (TS d73d71e1531b4649b363236998019221 not found in new config with opid_index 679) from {username='slave'} at 127.0.0.1:37186
I20240501 20:10:25.472712 31389 raft_consensus.cc:1261] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer 582c7a51c0ac43b4934078ebdf434485: Log matching property violated. Preceding OpId in replica: term: 1 index: 678. Preceding OpId from leader: term: 1 index: 679. (index mismatch)
I20240501 20:10:25.477353 31644 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.679: config changed from index 678 to 679, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 679 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.478688 3397 consensus_queue.cc:1035] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 679, Last known committed idx: 678, Time since last communication: 0.000s
I20240501 20:10:25.481969 31389 raft_consensus.cc:2932] T 7faad10e555240378c4deeb23188128a P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.679: config changed from index 678 to 679, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 679 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:25.501242 3414 tablet_replica.cc:330] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: stopping tablet replica
I20240501 20:10:25.501555 3414 raft_consensus.cc:2229] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.502040 3414 raft_consensus.cc:2258] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.504531 3414 ts_tablet_manager.cc:1875] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.512615 3414 ts_tablet_manager.cc:1888] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.678
I20240501 20:10:25.512856 3414 log.cc:1196] T 7faad10e555240378c4deeb23188128a P d73d71e1531b4649b363236998019221: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/wals/7faad10e555240378c4deeb23188128a
I20240501 20:10:25.514122 31102 catalog_manager.cc:4830] TS d73d71e1531b4649b363236998019221 (127.28.136.69:43505): tablet 7faad10e555240378c4deeb23188128a (table rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]) successfully deleted
I20240501 20:10:25.515256 2798 raft_consensus.cc:1203] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:25.539536 3425 ts_tablet_manager.cc:902] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
I20240501 20:10:25.574219 3425 tablet_copy_client.cc:308] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:43505
I20240501 20:10:25.575232 31800 tablet_copy_service.cc:140] P d73d71e1531b4649b363236998019221: Received BeginTabletCopySession request for tablet 87de1a5930394b2ca4ed6e99d01936d7 from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:53079)
I20240501 20:10:25.575423 31800 tablet_copy_service.cc:161] P d73d71e1531b4649b363236998019221: Beginning new tablet copy session on tablet 87de1a5930394b2ca4ed6e99d01936d7 from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:53079: session id = f27d0fb719e1420c8dbbc10ba0d318f9-87de1a5930394b2ca4ed6e99d01936d7
I20240501 20:10:25.578423 31800 tablet_copy_source_session.cc:205] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:25.582520 3425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87de1a5930394b2ca4ed6e99d01936d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.587728 3302 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=357 overwritten=0 applied=357 ignored=0} inserts{seen=1039 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:25.588132 3302 tablet_bootstrap.cc:492] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:25.588431 3302 ts_tablet_manager.cc:1367] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.542s user 0.383s sys 0.041s
I20240501 20:10:25.589237 3425 tablet_copy_client.cc:783] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:25.589160 3302 raft_consensus.cc:348] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:25.589704 3302 raft_consensus.cc:729] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:25.589752 3425 tablet_copy_client.cc:647] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:25.589959 3302 consensus_queue.cc:260] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 357, Last appended: 1.357, Last appended by leader: 357, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:25.590655 3302 ts_tablet_manager.cc:1398] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:10:25.591583 31669 tablet_copy_service.cc:342] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-e93ddb68383c43089544a6f0549c83e2 received from {username='slave'} at 127.28.136.72:53361
I20240501 20:10:25.591722 31669 tablet_copy_service.cc:433] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-e93ddb68383c43089544a6f0549c83e2 on tablet e93ddb68383c43089544a6f0549c83e2 with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:25.596411 3425 tablet_copy_client.cc:523] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:25.599766 3425 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:25.865207 3302 ts_tablet_manager.cc:902] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049)
I20240501 20:10:25.893321 3302 tablet_copy_client.cc:308] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:44049
I20240501 20:10:25.894879 31669 tablet_copy_service.cc:140] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Received BeginTabletCopySession request for tablet dda27412e67c4571bd8c5e86a65b787f from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:53361)
I20240501 20:10:25.895076 31669 tablet_copy_service.cc:161] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Beginning new tablet copy session on tablet dda27412e67c4571bd8c5e86a65b787f from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:53361: session id = f27d0fb719e1420c8dbbc10ba0d318f9-dda27412e67c4571bd8c5e86a65b787f
I20240501 20:10:25.897903 31669 tablet_copy_source_session.cc:205] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:25.899405 3302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda27412e67c4571bd8c5e86a65b787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.906862 3302 tablet_copy_client.cc:783] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:25.907212 3302 tablet_copy_client.cc:647] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:25.910411 3302 tablet_copy_client.cc:523] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:25.914286 3302 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:26.049432 2980 raft_consensus.cc:1050] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:26.050765 2980 consensus_queue.cc:237] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:26.067751 31256 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.068256 31644 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.069417 2798 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.069696 3209 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.072096 3210 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.072646 3210 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.104552 3210 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.109402 31255 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.116680 31385 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.116669 31114 catalog_manager.cc:5484] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index 2 to 3, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d73d71e1531b4649b363236998019221" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.117303 2980 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.118142 401 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.119153 2798 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.124394 3209 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.124363 31644 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.133782 401 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.137148 31385 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.169267 2798 raft_consensus.cc:1203] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.356->[1.357-1.357] Dedup: 1.357->[]
I20240501 20:10:26.253144 3302 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=301 overwritten=0 applied=301 ignored=0} inserts{seen=970 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:26.253834 3302 tablet_bootstrap.cc:492] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:26.254314 3302 ts_tablet_manager.cc:1367] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.340s user 0.303s sys 0.025s
I20240501 20:10:26.255438 3302 raft_consensus.cc:348] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:26.255888 3302 raft_consensus.cc:729] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:26.256232 3302 consensus_queue.cc:260] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 301, Last appended: 1.301, Last appended by leader: 301, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:26.256985 3302 ts_tablet_manager.cc:1398] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:26.258152 31669 tablet_copy_service.cc:342] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-dda27412e67c4571bd8c5e86a65b787f received from {username='slave'} at 127.28.136.72:53361
I20240501 20:10:26.258296 31669 tablet_copy_service.cc:433] P ce9dba79a8c94a7b87bf51e9ed4b5fe1: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-dda27412e67c4571bd8c5e86a65b787f on tablet dda27412e67c4571bd8c5e86a65b787f with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:26.350387 3425 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=693 overwritten=0 applied=693 ignored=0} inserts{seen=1365 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:26.350764 3425 tablet_bootstrap.cc:492] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:26.351033 3425 ts_tablet_manager.cc:1367] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.751s user 0.573s sys 0.073s
I20240501 20:10:26.351617 3425 raft_consensus.cc:348] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:26.351850 3425 raft_consensus.cc:729] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:26.352002 3425 consensus_queue.cc:260] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 693, Last appended: 1.693, Last appended by leader: 693, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 693 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:26.352546 3425 ts_tablet_manager.cc:1398] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 20:10:26.353343 31800 tablet_copy_service.cc:342] P d73d71e1531b4649b363236998019221: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-87de1a5930394b2ca4ed6e99d01936d7 received from {username='slave'} at 127.28.136.72:53079
I20240501 20:10:26.353490 31800 tablet_copy_service.cc:433] P d73d71e1531b4649b363236998019221: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-87de1a5930394b2ca4ed6e99d01936d7 on tablet 87de1a5930394b2ca4ed6e99d01936d7 with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:26.380311 2798 raft_consensus.cc:1203] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.300->[1.301-1.301] Dedup: 1.301->[]
I20240501 20:10:26.468199 2798 raft_consensus.cc:1203] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.692->[1.693-1.693] Dedup: 1.693->[]
I20240501 20:10:26.709707 31775 consensus_queue.cc:237] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:26.711987 31644 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:26.711987 2798 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:26.712544 31385 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:26.712673 401 raft_consensus.cc:1261] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:26.712663 3210 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.713268 3210 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.713559 3209 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.713804 3537 consensus_queue.cc:1035] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.717234 3166 raft_consensus.cc:1050] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:26.718035 3166 consensus_queue.cc:237] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 1.357, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:26.718189 3537 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.718292 401 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.720585 403 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240501 20:10:26.720619 2797 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240501 20:10:26.721172 3158 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 20:10:26.721560 2797 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.722080 3158 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 20:10:26.721859 31385 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.727268 31110 catalog_manager.cc:5484] T 750c72210b3240839bb156782aa1bdca P f27d0fb719e1420c8dbbc10ba0d318f9 reported cstate change: config changed from index 3 to 4, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "d73d71e1531b4649b363236998019221" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.729847 31909 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240501 20:10:26.731702 31775 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240501 20:10:26.731331 31644 raft_consensus.cc:2932] T 750c72210b3240839bb156782aa1bdca P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.731935 31385 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 1 index: 358. (index mismatch)
I20240501 20:10:26.732596 31102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 750c72210b3240839bb156782aa1bdca with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:26.733256 3166 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 20:10:26.733810 3166 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 20:10:26.745496 2804 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 20:10:26.747453 2804 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.358: config changed from index 357 to 358, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.749666 31775 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 357 to 358, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.752871 2798 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 357 to 358, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.754350 403 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 357 to 358, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.756498 31909 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 357 to 358, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.762584 31114 catalog_manager.cc:5484] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 357 to 358, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.766494 31385 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.358: config changed from index 357 to 358, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 358 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.781762 31644 consensus_queue.cc:237] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 1.358, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:26.784423 31775 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20240501 20:10:26.784798 31385 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20240501 20:10:26.785233 3158 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20240501 20:10:26.785768 3158 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20240501 20:10:26.787379 2798 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20240501 20:10:26.788185 31909 raft_consensus.cc:1261] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 358. Preceding OpId from leader: term: 1 index: 359. (index mismatch)
I20240501 20:10:26.788908 3166 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20240501 20:10:26.789602 31237 tablet_service.cc:1514] Processing DeleteTablet for tablet 750c72210b3240839bb156782aa1bdca with delete_type TABLET_DATA_TOMBSTONED (TS 341d3bb73a5c4e8f96176a281db4a60f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51884
I20240501 20:10:26.789769 3158 consensus_queue.cc:1035] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20240501 20:10:26.794319 3541 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.359: config changed from index 358 to 359, VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.798249 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e93ddb68383c43089544a6f0549c83e2 with cas_config_opid_index 358: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:26.798192 31775 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 358 to 359, VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.799044 31909 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 358 to 359, VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.799360 31110 catalog_manager.cc:5484] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 reported cstate change: config changed from index 358 to 359, VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) evicted. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:26.802414 2798 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 358 to 359, VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.805140 3554 tablet_replica.cc:330] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: stopping tablet replica
I20240501 20:10:26.805460 3554 raft_consensus.cc:2229] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.805817 3554 raft_consensus.cc:2258] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.805541 31385 raft_consensus.cc:2932] T e93ddb68383c43089544a6f0549c83e2 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.359: config changed from index 358 to 359, VOTER c42de85026a84ed7b4f462cf25433e01 (127.28.136.71) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.807535 3554 ts_tablet_manager.cc:1875] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.823298 3554 ts_tablet_manager.cc:1888] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:26.823458 3554 log.cc:1196] T 750c72210b3240839bb156782aa1bdca P 341d3bb73a5c4e8f96176a281db4a60f: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/750c72210b3240839bb156782aa1bdca
I20240501 20:10:26.824307 31104 catalog_manager.cc:4830] TS 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): tablet 750c72210b3240839bb156782aa1bdca (table TestTable [id=5168dc22b35340eb910bd1a9fe6598c9]) successfully deleted
I20240501 20:10:26.831650 387 tablet_service.cc:1514] Processing DeleteTablet for tablet e93ddb68383c43089544a6f0549c83e2 with delete_type TABLET_DATA_TOMBSTONED (TS c42de85026a84ed7b4f462cf25433e01 not found in new config with opid_index 359) from {username='slave'} at 127.0.0.1:50444
I20240501 20:10:26.832914 3558 tablet_replica.cc:330] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: stopping tablet replica
I20240501 20:10:26.833664 3558 raft_consensus.cc:2229] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.834156 3558 raft_consensus.cc:2258] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.836532 3558 ts_tablet_manager.cc:1875] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.843119 3558 ts_tablet_manager.cc:1888] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.358
I20240501 20:10:26.843262 3558 log.cc:1196] T e93ddb68383c43089544a6f0549c83e2 P c42de85026a84ed7b4f462cf25433e01: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal/wals/e93ddb68383c43089544a6f0549c83e2
I20240501 20:10:26.844372 31104 catalog_manager.cc:4830] TS c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749): tablet e93ddb68383c43089544a6f0549c83e2 (table rebalance_test_table_1 [id=ab3c75c3bb2748fcb14f718749a6bfbb]) successfully deleted
I20240501 20:10:26.848896 3158 raft_consensus.cc:1050] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:26.849848 3158 consensus_queue.cc:237] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:26.852514 403 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240501 20:10:26.853088 31775 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240501 20:10:26.853211 2798 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240501 20:10:26.853318 31909 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240501 20:10:26.853698 3166 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240501 20:10:26.853981 2804 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240501 20:10:26.854405 3541 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240501 20:10:26.856380 2804 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240501 20:10:26.865281 31255 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240501 20:10:26.865792 3570 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240501 20:10:26.868949 2804 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.302: config changed from index 301 to 302, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.869539 31255 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.874581 31110 catalog_manager.cc:5484] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index 301 to 302, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.875941 31775 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.876890 2798 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.877796 403 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.877949 31909 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } attrs { replace: true } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.889717 31644 consensus_queue.cc:237] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:26.891891 401 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240501 20:10:26.892028 31909 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240501 20:10:26.892310 31255 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240501 20:10:26.892784 2798 raft_consensus.cc:1261] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer ce9dba79a8c94a7b87bf51e9ed4b5fe1: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240501 20:10:26.892766 3541 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240501 20:10:26.893304 3166 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240501 20:10:26.894070 2804 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240501 20:10:26.895531 3166 consensus_queue.cc:1035] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240501 20:10:26.896788 3570 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.898813 2798 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.900439 31101 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dda27412e67c4571bd8c5e86a65b787f with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:26.900472 31909 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.900944 401 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.904460 31255 raft_consensus.cc:2932] T dda27412e67c4571bd8c5e86a65b787f P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.906308 31113 catalog_manager.cc:5484] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 reported cstate change: config changed from index 302 to 303, VOTER d73d71e1531b4649b363236998019221 (127.28.136.69) evicted. New cstate: current_term: 1 leader_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.913875 31759 tablet_service.cc:1514] Processing DeleteTablet for tablet dda27412e67c4571bd8c5e86a65b787f with delete_type TABLET_DATA_TOMBSTONED (TS d73d71e1531b4649b363236998019221 not found in new config with opid_index 303) from {username='slave'} at 127.0.0.1:37186
I20240501 20:10:26.914674 3488 raft_consensus.cc:1050] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:26.915601 3488 consensus_queue.cc:237] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 693, Committed index: 693, Last appended: 1.693, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:26.916167 3585 tablet_replica.cc:330] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: stopping tablet replica
I20240501 20:10:26.916404 3585 raft_consensus.cc:2229] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.916682 3585 raft_consensus.cc:2258] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.918203 3585 ts_tablet_manager.cc:1875] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.922904 2798 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 693. Preceding OpId from leader: term: 1 index: 694. (index mismatch)
I20240501 20:10:26.923085 31644 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 693. Preceding OpId from leader: term: 1 index: 694. (index mismatch)
I20240501 20:10:26.923787 31515 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 693. Preceding OpId from leader: term: 1 index: 694. (index mismatch)
I20240501 20:10:26.924180 3585 ts_tablet_manager.cc:1888] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.302
I20240501 20:10:26.924358 3585 log.cc:1196] T dda27412e67c4571bd8c5e86a65b787f P d73d71e1531b4649b363236998019221: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/wals/dda27412e67c4571bd8c5e86a65b787f
I20240501 20:10:26.924479 3537 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 694, Last known committed idx: 693, Time since last communication: 0.000s
I20240501 20:10:26.924896 31385 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 693. Preceding OpId from leader: term: 1 index: 694. (index mismatch)
I20240501 20:10:26.924870 3210 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 694, Last known committed idx: 693, Time since last communication: 0.000s
I20240501 20:10:26.925161 3488 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 694, Last known committed idx: 693, Time since last communication: 0.000s
I20240501 20:10:26.925556 31102 catalog_manager.cc:4830] TS d73d71e1531b4649b363236998019221 (127.28.136.69:43505): tablet dda27412e67c4571bd8c5e86a65b787f (table rebalance_test_table_2 [id=e358c69bd7b74e439c3651a47b25e3e7]) successfully deleted
I20240501 20:10:26.925932 31255 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 693. Preceding OpId from leader: term: 1 index: 694. (index mismatch)
I20240501 20:10:26.926370 3209 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 694, Last known committed idx: 693, Time since last communication: 0.000s
I20240501 20:10:26.926633 3537 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 694, Last known committed idx: 693, Time since last communication: 0.000s
I20240501 20:10:26.940776 3209 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 1 LEADER]: Committing config change with OpId 1.694: config changed from index 693 to 694, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.941888 31385 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.694: config changed from index 693 to 694, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.946476 2798 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.694: config changed from index 693 to 694, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.947824 31255 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.694: config changed from index 693 to 694, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.948904 31115 catalog_manager.cc:5484] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 reported cstate change: config changed from index 693 to 694, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d73d71e1531b4649b363236998019221" committed_config { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:26.960251 31515 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.694: config changed from index 693 to 694, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.959795 31644 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.694: config changed from index 693 to 694, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 694 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } attrs { replace: true } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.963793 31775 consensus_queue.cc:237] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 694, Committed index: 694, Last appended: 1.694, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 695 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:26.966905 2798 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 694. Preceding OpId from leader: term: 1 index: 695. (index mismatch)
I20240501 20:10:26.967128 31385 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 694. Preceding OpId from leader: term: 1 index: 695. (index mismatch)
I20240501 20:10:26.967482 3210 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 695, Last known committed idx: 694, Time since last communication: 0.000s
I20240501 20:10:26.967741 3537 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 695, Last known committed idx: 694, Time since last communication: 0.000s
I20240501 20:10:26.968261 31255 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 694. Preceding OpId from leader: term: 1 index: 695. (index mismatch)
I20240501 20:10:26.968629 31514 raft_consensus.cc:1261] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Refusing update from remote peer d73d71e1531b4649b363236998019221: Log matching property violated. Preceding OpId in replica: term: 1 index: 694. Preceding OpId from leader: term: 1 index: 695. (index mismatch)
I20240501 20:10:26.968941 3537 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 695, Last known committed idx: 694, Time since last communication: 0.000s
I20240501 20:10:26.969693 3537 consensus_queue.cc:1035] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 695, Last known committed idx: 694, Time since last communication: 0.000s
I20240501 20:10:26.971486 3210 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 [term 1 LEADER]: Committing config change with OpId 1.695: config changed from index 694 to 695, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 695 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.971997 31255 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.695: config changed from index 694 to 695, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 695 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.973995 2798 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.695: config changed from index 694 to 695, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 695 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.974038 31515 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P ec65c12cd0724cf9aa16fcd61bbedae4 [term 1 FOLLOWER]: Committing config change with OpId 1.695: config changed from index 694 to 695, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 695 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.975323 31102 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87de1a5930394b2ca4ed6e99d01936d7 with cas_config_opid_index 694: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:26.976656 31385 raft_consensus.cc:2932] T 87de1a5930394b2ca4ed6e99d01936d7 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Committing config change with OpId 1.695: config changed from index 694 to 695, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New config: { opid_index: 695 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.980393 31115 catalog_manager.cc:5484] T 87de1a5930394b2ca4ed6e99d01936d7 P f27d0fb719e1420c8dbbc10ba0d318f9 reported cstate change: config changed from index 694 to 695, VOTER ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "d73d71e1531b4649b363236998019221" committed_config { opid_index: 695 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:26.985138 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet 87de1a5930394b2ca4ed6e99d01936d7 with delete_type TABLET_DATA_TOMBSTONED (TS ce9dba79a8c94a7b87bf51e9ed4b5fe1 not found in new config with opid_index 695) from {username='slave'} at 127.0.0.1:44544
I20240501 20:10:26.986116 3600 tablet_replica.cc:330] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: stopping tablet replica
I20240501 20:10:26.987094 3600 raft_consensus.cc:2229] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.987587 3600 raft_consensus.cc:2258] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.989832 3600 ts_tablet_manager.cc:1875] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.996137 3600 ts_tablet_manager.cc:1888] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.694
I20240501 20:10:26.996241 3600 log.cc:1196] T 87de1a5930394b2ca4ed6e99d01936d7 P ce9dba79a8c94a7b87bf51e9ed4b5fe1: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/87de1a5930394b2ca4ed6e99d01936d7
I20240501 20:10:26.996932 31101 catalog_manager.cc:4830] TS ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): tablet 87de1a5930394b2ca4ed6e99d01936d7 (table rebalance_test_table_3 [id=f20b4eff405e4a3d85fddbfd4f9a2474]) successfully deleted
I20240501 20:10:27.130113 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:27.132638 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:27.133224 31237 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:27.137269 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:27.151880 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:27.153931 31367 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:27.203801 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:27.237322 2778 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:27.328285 31385 consensus_queue.cc:237] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 522, Committed index: 522, Last appended: 1.522, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:27.331583 31255 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 522. Preceding OpId from leader: term: 1 index: 523. (index mismatch)
I20240501 20:10:27.332376 3476 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
W20240501 20:10:27.349061 31322 consensus_peers.cc:485] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 -> Peer f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299): Couldn't send request to peer f27d0fb719e1420c8dbbc10ba0d318f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66e635d773d34c9bbf8b2b15c4611b40. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:27.356976 31644 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 522. Preceding OpId from leader: term: 1 index: 523. (index mismatch)
I20240501 20:10:27.357667 31909 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 522. Preceding OpId from leader: term: 1 index: 523. (index mismatch)
I20240501 20:10:27.358779 3644 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I20240501 20:10:27.359412 31775 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 522. Preceding OpId from leader: term: 1 index: 523. (index mismatch)
I20240501 20:10:27.359442 3644 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I20240501 20:10:27.361852 3476 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 523, Last known committed idx: 522, Time since last communication: 0.000s
I20240501 20:10:27.364657 3476 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Committing config change with OpId 1.523: config changed from index -1 to 523, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:27.365926 31644 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.523: config changed from index -1 to 523, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:27.369959 31255 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.523: config changed from index -1 to 523, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:27.370833 31113 catalog_manager.cc:5484] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 reported cstate change: config changed from index -1 to 523, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "db07c3a323eb4a3c919eaf9b19b24986" committed_config { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:27.379314 31775 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.523: config changed from index -1 to 523, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:27.384294 31909 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.523: config changed from index -1 to 523, NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) added. New config: { opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } } }
I20240501 20:10:27.947726 3668 ts_tablet_manager.cc:902] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: Initiating tablet copy from peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527)
I20240501 20:10:27.948813 3668 tablet_copy_client.cc:308] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:42527
I20240501 20:10:27.954428 31409 tablet_copy_service.cc:140] P db07c3a323eb4a3c919eaf9b19b24986: Received BeginTabletCopySession request for tablet 66e635d773d34c9bbf8b2b15c4611b40 from peer f27d0fb719e1420c8dbbc10ba0d318f9 ({username='slave'} at 127.28.136.72:33461)
I20240501 20:10:27.954567 31409 tablet_copy_service.cc:161] P db07c3a323eb4a3c919eaf9b19b24986: Beginning new tablet copy session on tablet 66e635d773d34c9bbf8b2b15c4611b40 from peer f27d0fb719e1420c8dbbc10ba0d318f9 at {username='slave'} at 127.28.136.72:33461: session id = f27d0fb719e1420c8dbbc10ba0d318f9-66e635d773d34c9bbf8b2b15c4611b40
I20240501 20:10:27.956579 31409 tablet_copy_source_session.cc:205] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:27.957969 3668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e635d773d34c9bbf8b2b15c4611b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.962487 3668 tablet_copy_client.cc:783] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:27.962731 3668 tablet_copy_client.cc:647] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:27.965490 3668 tablet_copy_client.cc:523] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:27.967905 3668 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap starting.
I20240501 20:10:28.425159 3668 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=523 overwritten=0 applied=523 ignored=0} inserts{seen=1195 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:28.425529 3668 tablet_bootstrap.cc:492] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: Bootstrap complete.
I20240501 20:10:28.425827 3668 ts_tablet_manager.cc:1367] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent bootstrapping tablet: real 0.458s user 0.439s sys 0.020s
I20240501 20:10:28.426580 3668 raft_consensus.cc:348] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:28.426872 3668 raft_consensus.cc:729] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f27d0fb719e1420c8dbbc10ba0d318f9, State: Initialized, Role: LEARNER
I20240501 20:10:28.427238 3668 consensus_queue.cc:260] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 523, Last appended: 1.523, Last appended by leader: 523, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 523 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: true } }
I20240501 20:10:28.428078 3668 ts_tablet_manager.cc:1398] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:28.428797 31409 tablet_copy_service.cc:342] P db07c3a323eb4a3c919eaf9b19b24986: Request end of tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-66e635d773d34c9bbf8b2b15c4611b40 received from {username='slave'} at 127.28.136.72:33461
I20240501 20:10:28.428920 31409 tablet_copy_service.cc:433] P db07c3a323eb4a3c919eaf9b19b24986: ending tablet copy session f27d0fb719e1420c8dbbc10ba0d318f9-66e635d773d34c9bbf8b2b15c4611b40 on tablet 66e635d773d34c9bbf8b2b15c4611b40 with peer f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:28.508904 2798 raft_consensus.cc:1203] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.522->[1.523-1.523] Dedup: 1.523->[]
I20240501 20:10:29.040486 3703 raft_consensus.cc:1050] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: attempting to promote NON_VOTER f27d0fb719e1420c8dbbc10ba0d318f9 to VOTER
I20240501 20:10:29.041411 3703 consensus_queue.cc:237] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 523, Committed index: 523, Last appended: 1.523, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:29.043747 2798 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 LEARNER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240501 20:10:29.044322 3692 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240501 20:10:29.044631 31256 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240501 20:10:29.044646 31644 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240501 20:10:29.045212 3692 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240501 20:10:29.046259 3692 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240501 20:10:29.049270 31775 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240501 20:10:29.049504 31909 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Refusing update from remote peer db07c3a323eb4a3c919eaf9b19b24986: Log matching property violated. Preceding OpId in replica: term: 1 index: 523. Preceding OpId from leader: term: 1 index: 524. (index mismatch)
I20240501 20:10:29.049970 3703 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240501 20:10:29.050333 3692 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 524, Last known committed idx: 523, Time since last communication: 0.000s
I20240501 20:10:29.053993 3692 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Committing config change with OpId 1.524: config changed from index 523 to 524, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:29.054701 2798 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:29.055374 31909 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:29.056036 31775 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:29.058472 31256 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:29.060704 31109 catalog_manager.cc:5484] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 reported cstate change: config changed from index 523 to 524, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db07c3a323eb4a3c919eaf9b19b24986" committed_config { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:29.063190 31644 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Committing config change with OpId 1.524: config changed from index 523 to 524, f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:29.107645 31385 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40"
dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35170
I20240501 20:10:29.107883 31385 raft_consensus.cc:595] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:29.327098 31385 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40"
dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35192
I20240501 20:10:29.327302 31385 raft_consensus.cc:595] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:29.476910 3705 raft_consensus.cc:979] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: : Instructing follower 341d3bb73a5c4e8f96176a281db4a60f to start an election
I20240501 20:10:29.477087 3705 raft_consensus.cc:1067] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Signalling peer 341d3bb73a5c4e8f96176a281db4a60f to start an election
I20240501 20:10:29.477895 31255 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40"
dest_uuid: "341d3bb73a5c4e8f96176a281db4a60f"
from {username='slave'} at 127.28.136.66:53803
I20240501 20:10:29.478139 31255 raft_consensus.cc:482] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:29.478248 31255 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:29.480278 31255 raft_consensus.cc:504] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:29.481472 31255 leader_election.cc:288] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 2 election: Requested vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299)
I20240501 20:10:29.481626 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 2 candidate_status { last_received { term: 1 index: 524 } } ignore_live_leader: true dest_uuid: "582c7a51c0ac43b4934078ebdf434485"
I20240501 20:10:29.481855 31909 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:29.481825 31775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 2 candidate_status { last_received { term: 1 index: 524 } } ignore_live_leader: true dest_uuid: "d73d71e1531b4649b363236998019221"
I20240501 20:10:29.482017 31775 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:29.482260 31644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 2 candidate_status { last_received { term: 1 index: 524 } } ignore_live_leader: true dest_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1"
I20240501 20:10:29.482466 31644 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:29.482538 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 2 candidate_status { last_received { term: 1 index: 524 } } ignore_live_leader: true dest_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9"
I20240501 20:10:29.482858 2798 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:29.484571 31909 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 2.
I20240501 20:10:29.484571 31775 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 2.
I20240501 20:10:29.485153 31644 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 2.
I20240501 20:10:29.485476 31385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "341d3bb73a5c4e8f96176a281db4a60f" candidate_term: 2 candidate_status { last_received { term: 1 index: 524 } } ignore_live_leader: true dest_uuid: "db07c3a323eb4a3c919eaf9b19b24986"
I20240501 20:10:29.485682 31385 raft_consensus.cc:3032] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:29.485752 31385 raft_consensus.cc:729] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: db07c3a323eb4a3c919eaf9b19b24986, State: Running, Role: LEADER
I20240501 20:10:29.486083 31385 consensus_queue.cc:260] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 524, Committed index: 524, Last appended: 1.524, Last appended by leader: 524, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:29.486646 31385 raft_consensus.cc:3037] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:29.487149 2798 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 2.
I20240501 20:10:29.487655 31191 leader_election.cc:302] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 341d3bb73a5c4e8f96176a281db4a60f, 582c7a51c0ac43b4934078ebdf434485, ce9dba79a8c94a7b87bf51e9ed4b5fe1, f27d0fb719e1420c8dbbc10ba0d318f9; no voters:
I20240501 20:10:29.488016 3490 raft_consensus.cc:2781] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:29.489223 31385 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d3bb73a5c4e8f96176a281db4a60f in term 2.
I20240501 20:10:29.490419 3490 raft_consensus.cc:686] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 2 LEADER]: Becoming Leader. State: Replica: 341d3bb73a5c4e8f96176a281db4a60f, State: Running, Role: LEADER
I20240501 20:10:29.490772 3490 consensus_queue.cc:237] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 524, Committed index: 524, Last appended: 1.524, Last appended by leader: 524, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:29.494400 31115 catalog_manager.cc:5484] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: term changed from 1 to 2, leader changed from db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) to 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65). New cstate: current_term: 2 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 524 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:29.880102 2798 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 2 index: 525. (index mismatch)
I20240501 20:10:29.880676 3490 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240501 20:10:29.932097 31644 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 2 index: 525. (index mismatch)
I20240501 20:10:29.932603 3490 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240501 20:10:29.980235 31909 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 2 index: 525. (index mismatch)
I20240501 20:10:29.980795 3490 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240501 20:10:29.986066 31775 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 2 index: 525. (index mismatch)
I20240501 20:10:29.986215 31385 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 1 index: 524. Preceding OpId from leader: term: 2 index: 525. (index mismatch)
I20240501 20:10:29.986765 3490 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240501 20:10:29.987102 3737 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 525, Last known committed idx: 524, Time since last communication: 0.000s
I20240501 20:10:30.596887 31255 consensus_queue.cc:237] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 525, Committed index: 525, Last appended: 2.525, Last appended by leader: 524, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:30.599673 31909 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 2 index: 525. Preceding OpId from leader: term: 2 index: 526. (index mismatch)
I20240501 20:10:30.599752 31644 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 2 index: 525. Preceding OpId from leader: term: 2 index: 526. (index mismatch)
I20240501 20:10:30.599790 2798 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 2 index: 525. Preceding OpId from leader: term: 2 index: 526. (index mismatch)
I20240501 20:10:30.600418 31775 raft_consensus.cc:1261] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 2 FOLLOWER]: Refusing update from remote peer 341d3bb73a5c4e8f96176a281db4a60f: Log matching property violated. Preceding OpId in replica: term: 2 index: 525. Preceding OpId from leader: term: 2 index: 526. (index mismatch)
I20240501 20:10:30.600488 3760 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 526, Last known committed idx: 525, Time since last communication: 0.000s
I20240501 20:10:30.601167 3760 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 526, Last known committed idx: 525, Time since last communication: 0.000s
I20240501 20:10:30.601948 3737 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 526, Last known committed idx: 525, Time since last communication: 0.000s
I20240501 20:10:30.602766 3758 consensus_queue.cc:1035] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 526, Last known committed idx: 525, Time since last communication: 0.000s
I20240501 20:10:30.604450 3758 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f [term 2 LEADER]: Committing config change with OpId 2.526: config changed from index 524 to 526, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:30.607723 31104 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 66e635d773d34c9bbf8b2b15c4611b40 with cas_config_opid_index 524: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:30.608314 31644 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 [term 2 FOLLOWER]: Committing config change with OpId 2.526: config changed from index 524 to 526, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:30.609155 31113 catalog_manager.cc:5484] T 66e635d773d34c9bbf8b2b15c4611b40 P 341d3bb73a5c4e8f96176a281db4a60f reported cstate change: config changed from index 524 to 526, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New cstate: current_term: 2 leader_uuid: "341d3bb73a5c4e8f96176a281db4a60f" committed_config { opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:30.610508 31909 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 2 FOLLOWER]: Committing config change with OpId 2.526: config changed from index 524 to 526, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:30.610531 31775 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P d73d71e1531b4649b363236998019221 [term 2 FOLLOWER]: Committing config change with OpId 2.526: config changed from index 524 to 526, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:30.612013 2798 raft_consensus.cc:2932] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 2 FOLLOWER]: Committing config change with OpId 2.526: config changed from index 524 to 526, VOTER db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66) evicted. New config: { opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } } }
I20240501 20:10:30.619099 31367 tablet_service.cc:1514] Processing DeleteTablet for tablet 66e635d773d34c9bbf8b2b15c4611b40 with delete_type TABLET_DATA_TOMBSTONED (TS db07c3a323eb4a3c919eaf9b19b24986 not found in new config with opid_index 526) from {username='slave'} at 127.0.0.1:51676
I20240501 20:10:30.619724 3786 tablet_replica.cc:330] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: stopping tablet replica
I20240501 20:10:30.620025 3786 raft_consensus.cc:2229] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:30.620308 3786 raft_consensus.cc:2258] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:30.622121 3786 ts_tablet_manager.cc:1875] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:30.626899 3786 ts_tablet_manager.cc:1888] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.525
I20240501 20:10:30.627027 3786 log.cc:1196] T 66e635d773d34c9bbf8b2b15c4611b40 P db07c3a323eb4a3c919eaf9b19b24986: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/66e635d773d34c9bbf8b2b15c4611b40
I20240501 20:10:30.627774 31104 catalog_manager.cc:4830] TS db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): tablet 66e635d773d34c9bbf8b2b15c4611b40 (table rebalance_test_table_4 [id=0cec196ed65049adb8e6a6690675a13e]) successfully deleted
I20240501 20:10:30.918341 31237 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:30.944269 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:30.945148 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:30.970516 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:30.971465 31367 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:30.988255 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:31.029997 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:31.074198 2778 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:31.261741 31237 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:31.266172 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:31.293848 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:31.296036 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:31.325162 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:31.334026 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:31.334779 2778 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:31.342396 31367 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:31.707849 31237 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:31.716683 31888 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:31.717852 31628 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:31.730167 387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:31.740242 31759 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:31.746686 31498 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:31.755756 31367 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:31.774827 2778 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
18e54ea4106c49c48e2a2a461e4657fd | 127.28.136.126:46583 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.28.136.84:37127 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
341d3bb73a5c4e8f96176a281db4a60f | 127.28.136.65:33401 | HEALTHY | <none> | 3 | 0
582c7a51c0ac43b4934078ebdf434485 | 127.28.136.70:37941 | HEALTHY | <none> | 2 | 0
c42de85026a84ed7b4f462cf25433e01 | 127.28.136.71:35749 | HEALTHY | <none> | 0 | 0
ce9dba79a8c94a7b87bf51e9ed4b5fe1 | 127.28.136.68:44049 | HEALTHY | <none> | 4 | 0
d73d71e1531b4649b363236998019221 | 127.28.136.69:43505 | HEALTHY | <none> | 2 | 0
db07c3a323eb4a3c919eaf9b19b24986 | 127.28.136.66:42527 | HEALTHY | <none> | 2 | 0
ec65c12cd0724cf9aa16fcd61bbedae4 | 127.28.136.67:42361 | HEALTHY | <none> | 3 | 0
f27d0fb719e1420c8dbbc10ba0d318f9 | 127.28.136.72:45299 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.28.136.65 | experimental | 127.28.136.65:33401
local_ip_for_outbound_sockets | 127.28.136.66 | experimental | 127.28.136.66:42527
local_ip_for_outbound_sockets | 127.28.136.67 | experimental | 127.28.136.67:42361
local_ip_for_outbound_sockets | 127.28.136.68 | experimental | 127.28.136.68:44049
local_ip_for_outbound_sockets | 127.28.136.69 | experimental | 127.28.136.69:43505
local_ip_for_outbound_sockets | 127.28.136.70 | experimental | 127.28.136.70:37941
local_ip_for_outbound_sockets | 127.28.136.71 | experimental | 127.28.136.71:35749
local_ip_for_outbound_sockets | 127.28.136.72 | experimental | 127.28.136.72:45299
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.136.65:33401
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.136.66:42527
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.136.67:42361
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.28.136.68:44049
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.136.69:43505
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.136.70:37941
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.28.136.71:35749
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.28.136.72:45299
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.28.136.84:37127 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:10:31.862447 29217 log_verifier.cc:126] Checking tablet 2c9d9115e89c437ca7b962201115f087
I20240501 20:10:32.137923 29217 log_verifier.cc:177] Verified matching terms for 272 ops in tablet 2c9d9115e89c437ca7b962201115f087
I20240501 20:10:32.138513 29217 log_verifier.cc:126] Checking tablet 37fbd40e4b9a4d75994b3e138c4506d8
I20240501 20:10:32.588241 29217 log_verifier.cc:177] Verified matching terms for 503 ops in tablet 37fbd40e4b9a4d75994b3e138c4506d8
I20240501 20:10:32.589440 29217 log_verifier.cc:126] Checking tablet 66e635d773d34c9bbf8b2b15c4611b40
I20240501 20:10:32.995358 29217 log_verifier.cc:177] Verified matching terms for 526 ops in tablet 66e635d773d34c9bbf8b2b15c4611b40
I20240501 20:10:32.996820 29217 log_verifier.cc:126] Checking tablet 750c72210b3240839bb156782aa1bdca
I20240501 20:10:33.026376 29217 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 750c72210b3240839bb156782aa1bdca
I20240501 20:10:33.026490 29217 log_verifier.cc:126] Checking tablet 791393d238c3413d9472eecffbf1438a
I20240501 20:10:33.387949 29217 log_verifier.cc:177] Verified matching terms for 389 ops in tablet 791393d238c3413d9472eecffbf1438a
I20240501 20:10:33.388904 29217 log_verifier.cc:126] Checking tablet 7faad10e555240378c4deeb23188128a
I20240501 20:10:33.951287 29217 log_verifier.cc:177] Verified matching terms for 679 ops in tablet 7faad10e555240378c4deeb23188128a
I20240501 20:10:33.952880 29217 log_verifier.cc:126] Checking tablet 87de1a5930394b2ca4ed6e99d01936d7
I20240501 20:10:34.560524 29217 log_verifier.cc:177] Verified matching terms for 695 ops in tablet 87de1a5930394b2ca4ed6e99d01936d7
I20240501 20:10:34.563377 29217 log_verifier.cc:126] Checking tablet 90512cd61a6647c4beb15ffcdd488da3
I20240501 20:10:34.997143 29217 log_verifier.cc:177] Verified matching terms for 546 ops in tablet 90512cd61a6647c4beb15ffcdd488da3
I20240501 20:10:34.998307 29217 log_verifier.cc:126] Checking tablet a3b0a8894c0f46c683461a1857ff27ff
I20240501 20:10:35.369642 29217 log_verifier.cc:177] Verified matching terms for 392 ops in tablet a3b0a8894c0f46c683461a1857ff27ff
I20240501 20:10:35.370534 29217 log_verifier.cc:126] Checking tablet abdd2df68c774be1ab6e39d9d5dbf563
I20240501 20:10:35.888312 29217 log_verifier.cc:177] Verified matching terms for 607 ops in tablet abdd2df68c774be1ab6e39d9d5dbf563
I20240501 20:10:35.889762 29217 log_verifier.cc:126] Checking tablet afb53b647def417c8c9e2378dbeb5dd9
I20240501 20:10:36.372045 29217 log_verifier.cc:177] Verified matching terms for 605 ops in tablet afb53b647def417c8c9e2378dbeb5dd9
I20240501 20:10:36.373373 29217 log_verifier.cc:126] Checking tablet d3b0fefa2f5248f49118f7fb59671b6c
I20240501 20:10:36.860505 29217 log_verifier.cc:177] Verified matching terms for 571 ops in tablet d3b0fefa2f5248f49118f7fb59671b6c
I20240501 20:10:36.861826 29217 log_verifier.cc:126] Checking tablet dda27412e67c4571bd8c5e86a65b787f
I20240501 20:10:37.152997 29217 log_verifier.cc:177] Verified matching terms for 303 ops in tablet dda27412e67c4571bd8c5e86a65b787f
I20240501 20:10:37.153788 29217 log_verifier.cc:126] Checking tablet dfe38cf1f2ef48e4adca9910e82e2d9e
I20240501 20:10:37.442960 29217 log_verifier.cc:177] Verified matching terms for 294 ops in tablet dfe38cf1f2ef48e4adca9910e82e2d9e
I20240501 20:10:37.443678 29217 log_verifier.cc:126] Checking tablet e93ddb68383c43089544a6f0549c83e2
I20240501 20:10:37.785044 29217 log_verifier.cc:177] Verified matching terms for 359 ops in tablet e93ddb68383c43089544a6f0549c83e2
I20240501 20:10:37.785914 29217 log_verifier.cc:126] Checking tablet ef4b7168ca5f4b2ba49ec83e57b48009
I20240501 20:10:38.369087 29217 log_verifier.cc:177] Verified matching terms for 719 ops in tablet ef4b7168ca5f4b2ba49ec83e57b48009
I20240501 20:10:38.377581 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 31176
I20240501 20:10:38.388860 31298 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:38.696686 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 31176
W20240501 20:10:38.741734 2731 connection.cc:535] server connection from 127.28.136.65:35171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:38.742492 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 31307
W20240501 20:10:38.742450 31585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.136.65:33401 (error 108)
W20240501 20:10:38.744699 31585 consensus_peers.cc:485] T dda27412e67c4571bd8c5e86a65b787f P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Couldn't send request to peer 341d3bb73a5c4e8f96176a281db4a60f. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.748929 31429 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:38.836391 31585 consensus_peers.cc:485] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Couldn't send request to peer 341d3bb73a5c4e8f96176a281db4a60f. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:38.857558 31846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
W20240501 20:10:38.858925 31846 consensus_peers.cc:485] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 -> Peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Couldn't send request to peer 341d3bb73a5c4e8f96176a281db4a60f. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:38.947953 31455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
W20240501 20:10:38.949257 31455 consensus_peers.cc:485] T ef4b7168ca5f4b2ba49ec83e57b48009 P ec65c12cd0724cf9aa16fcd61bbedae4 -> Peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Couldn't send request to peer 341d3bb73a5c4e8f96176a281db4a60f. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:38.953971 31715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240501 20:10:38.955073 31715 consensus_peers.cc:485] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 -> Peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Couldn't send request to peer 341d3bb73a5c4e8f96176a281db4a60f. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.043923 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 31307
W20240501 20:10:39.089169 337 connection.cc:535] server connection from 127.28.136.66:49659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.089249 31452 connection.cc:535] server connection from 127.28.136.66:51319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.089875 31846 connection.cc:535] server connection from 127.28.136.66:49877 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.089993 31102 connection.cc:535] server connection from 127.28.136.66:55935 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:39.090664 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 31437
I20240501 20:10:39.096215 31558 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:39.156428 31846 consensus_peers.cc:485] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 -> Peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Couldn't send request to peer db07c3a323eb4a3c919eaf9b19b24986. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.66:42527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.261255 31715 consensus_peers.cc:485] T 750c72210b3240839bb156782aa1bdca P d73d71e1531b4649b363236998019221 -> Peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Couldn't send request to peer db07c3a323eb4a3c919eaf9b19b24986. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.66:42527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.300818 31585 consensus_peers.cc:485] T e93ddb68383c43089544a6f0549c83e2 P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Couldn't send request to peer db07c3a323eb4a3c919eaf9b19b24986. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.66:42527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.359511 31715 consensus_peers.cc:485] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 -> Peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Couldn't send request to peer db07c3a323eb4a3c919eaf9b19b24986. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.66:42527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.399961 31585 consensus_peers.cc:485] T d3b0fefa2f5248f49118f7fb59671b6c P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Couldn't send request to peer db07c3a323eb4a3c919eaf9b19b24986. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.66:42527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.402417 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 31437
W20240501 20:10:39.449276 2733 connection.cc:535] server connection from 127.28.136.67:47411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.449659 31713 connection.cc:535] client connection to 127.28.136.67:42361 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.449710 31844 connection.cc:535] server connection from 127.28.136.67:58217 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:39.450554 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 31567
W20240501 20:10:39.450985 31583 consensus_peers.cc:485] T dfe38cf1f2ef48e4adca9910e82e2d9e P ce9dba79a8c94a7b87bf51e9ed4b5fe1 -> Peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Couldn't send request to peer ec65c12cd0724cf9aa16fcd61bbedae4. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.451121 31713 consensus_peers.cc:485] T 87de1a5930394b2ca4ed6e99d01936d7 P d73d71e1531b4649b363236998019221 -> Peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Couldn't send request to peer ec65c12cd0724cf9aa16fcd61bbedae4. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.456502 31688 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:39.463969 31844 consensus_peers.cc:485] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 -> Peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Couldn't send request to peer ec65c12cd0724cf9aa16fcd61bbedae4. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.616217 3951 raft_consensus.cc:482] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 341d3bb73a5c4e8f96176a281db4a60f)
I20240501 20:10:39.616484 3951 raft_consensus.cc:504] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:10:39.618274 3951 leader_election.cc:288] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
W20240501 20:10:39.619663 339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
W20240501 20:10:39.629071 339 leader_election.cc:334] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
I20240501 20:10:39.629046 31775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb53b647def417c8c9e2378dbeb5dd9" candidate_uuid: "c42de85026a84ed7b4f462cf25433e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 605 } } ignore_live_leader: false dest_uuid: "d73d71e1531b4649b363236998019221" is_pre_election: true
W20240501 20:10:39.629470 337 leader_election.cc:334] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111)
I20240501 20:10:39.629833 337 leader_election.cc:302] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: c42de85026a84ed7b4f462cf25433e01; no voters: 341d3bb73a5c4e8f96176a281db4a60f, d73d71e1531b4649b363236998019221, ec65c12cd0724cf9aa16fcd61bbedae4
I20240501 20:10:39.630142 3951 raft_consensus.cc:2726] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:39.758023 3951 raft_consensus.cc:482] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 341d3bb73a5c4e8f96176a281db4a60f)
I20240501 20:10:39.758260 3951 raft_consensus.cc:504] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:39.759495 3951 leader_election.cc:288] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299)
W20240501 20:10:39.761874 339 leader_election.cc:334] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
W20240501 20:10:39.762431 339 leader_election.cc:334] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Network error: Client connection negotiation failed: client connection to 127.28.136.66:42527: connect: Connection refused (error 111)
W20240501 20:10:39.762630 337 leader_election.cc:334] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111)
I20240501 20:10:39.762748 337 leader_election.cc:302] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: c42de85026a84ed7b4f462cf25433e01; no voters: 341d3bb73a5c4e8f96176a281db4a60f, db07c3a323eb4a3c919eaf9b19b24986, ec65c12cd0724cf9aa16fcd61bbedae4
I20240501 20:10:39.763410 3951 raft_consensus.cc:2726] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:39.768807 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 31567
I20240501 20:10:39.774364 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "c42de85026a84ed7b4f462cf25433e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 392 } } ignore_live_leader: false dest_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" is_pre_election: true
W20240501 20:10:39.815055 339 connection.cc:535] server connection from 127.28.136.68:40031 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:39.815611 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 31697
W20240501 20:10:39.816485 336 leader_election.cc:334] T afb53b647def417c8c9e2378dbeb5dd9 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): Network error: Client connection negotiation failed: client connection to 127.28.136.68:44049: connect: Connection refused (error 111)
W20240501 20:10:39.816648 31843 consensus_peers.cc:485] T 7faad10e555240378c4deeb23188128a P 582c7a51c0ac43b4934078ebdf434485 -> Peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): Couldn't send request to peer ce9dba79a8c94a7b87bf51e9ed4b5fe1. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.68:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.816787 31843 consensus_peers.cc:485] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 -> Peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): Couldn't send request to peer ce9dba79a8c94a7b87bf51e9ed4b5fe1. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.68:44049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.821053 31819 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:39.825345 3945 raft_consensus.cc:482] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 341d3bb73a5c4e8f96176a281db4a60f)
I20240501 20:10:39.825532 3945 raft_consensus.cc:504] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:39.826583 3945 leader_election.cc:288] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299)
I20240501 20:10:39.827319 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "582c7a51c0ac43b4934078ebdf434485" candidate_term: 3 candidate_status { last_received { term: 2 index: 526 } } ignore_live_leader: false dest_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" is_pre_election: true
W20240501 20:10:39.828379 31846 leader_election.cc:334] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
W20240501 20:10:39.828589 31843 leader_election.cc:334] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): Network error: Client connection negotiation failed: client connection to 127.28.136.68:44049: connect: Connection refused (error 111)
I20240501 20:10:39.828709 31843 leader_election.cc:302] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 582c7a51c0ac43b4934078ebdf434485; no voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:39.829008 3945 raft_consensus.cc:2726] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:10:40.098445 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 31697
W20240501 20:10:40.145159 31844 connection.cc:535] client connection to 127.28.136.69:43505 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:40.145772 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 31828
W20240501 20:10:40.146404 31844 leader_election.cc:334] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d71e1531b4649b363236998019221 (127.28.136.69:43505): Network error: Client connection negotiation failed: client connection to 127.28.136.69:43505: connect: Connection refused (error 111)
W20240501 20:10:40.146857 31844 consensus_peers.cc:485] T 791393d238c3413d9472eecffbf1438a P 582c7a51c0ac43b4934078ebdf434485 -> Peer d73d71e1531b4649b363236998019221 (127.28.136.69:43505): Couldn't send request to peer d73d71e1531b4649b363236998019221. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.69:43505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.151257 31950 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:40.192791 3970 raft_consensus.cc:482] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 341d3bb73a5c4e8f96176a281db4a60f)
I20240501 20:10:40.193042 3970 raft_consensus.cc:504] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 392 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:40.194959 3970 leader_election.cc:288] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749)
W20240501 20:10:40.197083 2734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
I20240501 20:10:40.197598 3970 raft_consensus.cc:482] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 341d3bb73a5c4e8f96176a281db4a60f)
I20240501 20:10:40.197825 3970 raft_consensus.cc:504] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 526 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:40.198990 3970 leader_election.cc:288] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), d73d71e1531b4649b363236998019221 (127.28.136.69:43505)
W20240501 20:10:40.199620 2734 leader_election.cc:334] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
W20240501 20:10:40.201355 2734 leader_election.cc:334] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Network error: Client connection negotiation failed: client connection to 127.28.136.66:42527: connect: Connection refused (error 111)
W20240501 20:10:40.202437 2734 leader_election.cc:334] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
W20240501 20:10:40.202517 2732 leader_election.cc:334] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111)
W20240501 20:10:40.202615 2731 leader_election.cc:334] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): Network error: Client connection negotiation failed: client connection to 127.28.136.68:44049: connect: Connection refused (error 111)
I20240501 20:10:40.202688 2732 leader_election.cc:302] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: f27d0fb719e1420c8dbbc10ba0d318f9; no voters: 341d3bb73a5c4e8f96176a281db4a60f, db07c3a323eb4a3c919eaf9b19b24986, ec65c12cd0724cf9aa16fcd61bbedae4
I20240501 20:10:40.203090 3970 raft_consensus.cc:2726] T a3b0a8894c0f46c683461a1857ff27ff P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.204073 2732 leader_election.cc:334] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d73d71e1531b4649b363236998019221 (127.28.136.69:43505): Network error: Client connection negotiation failed: client connection to 127.28.136.69:43505: connect: Connection refused (error 111)
I20240501 20:10:40.204213 2732 leader_election.cc:302] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: f27d0fb719e1420c8dbbc10ba0d318f9; no voters: 341d3bb73a5c4e8f96176a281db4a60f, ce9dba79a8c94a7b87bf51e9ed4b5fe1, d73d71e1531b4649b363236998019221
I20240501 20:10:40.204481 3970 raft_consensus.cc:2726] T 66e635d773d34c9bbf8b2b15c4611b40 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:10:40.206627 403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3b0a8894c0f46c683461a1857ff27ff" candidate_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 392 } } ignore_live_leader: false dest_uuid: "c42de85026a84ed7b4f462cf25433e01" is_pre_election: true
I20240501 20:10:40.206928 403 raft_consensus.cc:2454] T a3b0a8894c0f46c683461a1857ff27ff P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27d0fb719e1420c8dbbc10ba0d318f9 in term 1.
I20240501 20:10:40.424188 31909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e635d773d34c9bbf8b2b15c4611b40" candidate_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" candidate_term: 3 candidate_status { last_received { term: 2 index: 526 } } ignore_live_leader: false dest_uuid: "582c7a51c0ac43b4934078ebdf434485" is_pre_election: true
I20240501 20:10:40.424465 31909 raft_consensus.cc:2454] T 66e635d773d34c9bbf8b2b15c4611b40 P 582c7a51c0ac43b4934078ebdf434485 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27d0fb719e1420c8dbbc10ba0d318f9 in term 2.
I20240501 20:10:40.430047 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 31828
I20240501 20:10:40.471693 3981 raft_consensus.cc:482] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec65c12cd0724cf9aa16fcd61bbedae4)
I20240501 20:10:40.471872 3981 raft_consensus.cc:504] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:40.472802 3981 leader_election.cc:288] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), f27d0fb719e1420c8dbbc10ba0d318f9 (127.28.136.72:45299)
I20240501 20:10:40.474359 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fbd40e4b9a4d75994b3e138c4506d8" candidate_uuid: "c42de85026a84ed7b4f462cf25433e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 503 } } ignore_live_leader: false dest_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" is_pre_election: true
I20240501 20:10:40.475857 3981 raft_consensus.cc:482] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec65c12cd0724cf9aa16fcd61bbedae4)
I20240501 20:10:40.476004 3981 raft_consensus.cc:504] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
W20240501 20:10:40.476111 337 leader_election.cc:334] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111)
W20240501 20:10:40.476322 336 leader_election.cc:334] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): Network error: Client connection negotiation failed: client connection to 127.28.136.68:44049: connect: Connection refused (error 111)
I20240501 20:10:40.476433 336 leader_election.cc:302] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: c42de85026a84ed7b4f462cf25433e01; no voters: ce9dba79a8c94a7b87bf51e9ed4b5fe1, ec65c12cd0724cf9aa16fcd61bbedae4, f27d0fb719e1420c8dbbc10ba0d318f9
I20240501 20:10:40.476884 3981 leader_election.cc:288] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941)
I20240501 20:10:40.477049 3981 raft_consensus.cc:2726] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.478303 339 leader_election.cc:334] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401): Network error: Client connection negotiation failed: client connection to 127.28.136.65:33401: connect: Connection refused (error 111)
W20240501 20:10:40.478673 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:40.472900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.136.70:37941 (local address 127.28.136.71:46923)
0501 20:10:40.473315 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:40.473334 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:40.473456 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:40.478408 (+ 4952us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.136.70:37941: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":213,"threads_started":1}
W20240501 20:10:40.479374 337 leader_election.cc:334] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d73d71e1531b4649b363236998019221 (127.28.136.69:43505): Network error: Client connection negotiation failed: client connection to 127.28.136.69:43505: connect: Connection refused (error 111)
W20240501 20:10:40.479537 337 leader_election.cc:334] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111)
I20240501 20:10:40.479655 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 320
I20240501 20:10:40.479674 337 leader_election.cc:302] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: c42de85026a84ed7b4f462cf25433e01; no voters: 341d3bb73a5c4e8f96176a281db4a60f, d73d71e1531b4649b363236998019221, ec65c12cd0724cf9aa16fcd61bbedae4
W20240501 20:10:40.479815 336 leader_election.cc:334] T 37fbd40e4b9a4d75994b3e138c4506d8 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): Network error: Client connection negotiation failed: client connection to 127.28.136.70:37941: connect: Connection refused (error 111)
I20240501 20:10:40.479952 3986 raft_consensus.cc:2726] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.480119 336 leader_election.cc:334] T ef4b7168ca5f4b2ba49ec83e57b48009 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941): Network error: Client connection negotiation failed: client connection to 127.28.136.70:37941: connect: Connection refused (error 111)
I20240501 20:10:40.484515 447 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:40.491039 3970 raft_consensus.cc:482] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce9dba79a8c94a7b87bf51e9ed4b5fe1)
I20240501 20:10:40.491261 3970 raft_consensus.cc:504] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:40.492273 3970 leader_election.cc:288] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049), c42de85026a84ed7b4f462cf25433e01 (127.28.136.71:35749)
W20240501 20:10:40.493942 2732 leader_election.cc:334] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361): Network error: Client connection negotiation failed: client connection to 127.28.136.67:42361: connect: Connection refused (error 111)
W20240501 20:10:40.494102 2731 leader_election.cc:334] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ce9dba79a8c94a7b87bf51e9ed4b5fe1 (127.28.136.68:44049): Network error: Client connection negotiation failed: client connection to 127.28.136.68:44049: connect: Connection refused (error 111)
W20240501 20:10:40.494285 2734 leader_election.cc:334] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527): Network error: Client connection negotiation failed: client connection to 127.28.136.66:42527: connect: Connection refused (error 111)
I20240501 20:10:40.494412 2734 leader_election.cc:302] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: f27d0fb719e1420c8dbbc10ba0d318f9; no voters: ce9dba79a8c94a7b87bf51e9ed4b5fe1, db07c3a323eb4a3c919eaf9b19b24986, ec65c12cd0724cf9aa16fcd61bbedae4
I20240501 20:10:40.494659 3970 raft_consensus.cc:2726] T dfe38cf1f2ef48e4adca9910e82e2d9e P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.740572 3981 raft_consensus.cc:482] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec65c12cd0724cf9aa16fcd61bbedae4)
I20240501 20:10:40.740967 403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe38cf1f2ef48e4adca9910e82e2d9e" candidate_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 294 } } ignore_live_leader: false dest_uuid: "c42de85026a84ed7b4f462cf25433e01" is_pre_election: true
I20240501 20:10:40.740700 3986 raft_consensus.cc:482] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce9dba79a8c94a7b87bf51e9ed4b5fe1)
I20240501 20:10:40.741279 3981 raft_consensus.cc:504] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:10:40.741494 3986 raft_consensus.cc:504] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 294 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:40.742026 403 raft_consensus.cc:2454] T dfe38cf1f2ef48e4adca9910e82e2d9e P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f27d0fb719e1420c8dbbc10ba0d318f9 in term 1.
I20240501 20:10:40.742324 3981 leader_election.cc:288] T abdd2df68c774be1ab6e39d9d5dbf563 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), 582c7a51c0ac43b4934078ebdf434485 (127.28.136.70:37941), db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527)
I20240501 20:10:40.742465 3981 raft_consensus.cc:482] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader db07c3a323eb4a3c919eaf9b19b24986)
I20240501 20:10:40.742600 3981 raft_consensus.cc:504] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "db07c3a323eb4a3c919eaf9b19b24986" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 42527 } } peers { permanent_uuid: "d73d71e1531b4649b363236998019221" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 43505 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } }
I20240501 20:10:40.743361 3981 leader_election.cc:288] T 90512cd61a6647c4beb15ffcdd488da3 P c42de85026a84ed7b4f462cf25433e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db07c3a323eb4a3c919eaf9b19b24986 (127.28.136.66:42527), d73d71e1531b4649b363236998019221 (127.28.136.69:43505), ec65c12cd0724cf9aa16fcd61bbedae4 (127.28.136.67:42361), 341d3bb73a5c4e8f96176a281db4a60f (127.28.136.65:33401)
I20240501 20:10:40.744979 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 320
I20240501 20:10:40.788482 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 2715
I20240501 20:10:40.793985 2839 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:40.974283 3970 raft_consensus.cc:482] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec65c12cd0724cf9aa16fcd61bbedae4)
I20240501 20:10:40.974591 3991 raft_consensus.cc:482] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ce9dba79a8c94a7b87bf51e9ed4b5fe1)
I20240501 20:10:40.974557 3970 raft_consensus.cc:504] T 37fbd40e4b9a4d75994b3e138c4506d8 P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "ec65c12cd0724cf9aa16fcd61bbedae4" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 42361 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:40.974848 3991 raft_consensus.cc:504] T dda27412e67c4571bd8c5e86a65b787f P f27d0fb719e1420c8dbbc10ba0d318f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "341d3bb73a5c4e8f96176a281db4a60f" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 33401 } } peers { permanent_uuid: "582c7a51c0ac43b4934078ebdf434485" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 37941 } } peers { permanent_uuid: "ce9dba79a8c94a7b87bf51e9ed4b5fe1" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 44049 } } peers { permanent_uuid: "c42de85026a84ed7b4f462cf25433e01" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 35749 } attrs { promote: false } } peers { permanent_uuid: "f27d0fb719e1420c8dbbc10ba0d318f9" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 45299 } attrs { promote: false } }
I20240501 20:10:40.975147 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 2715
I20240501 20:10:41.000525 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 31086
I20240501 20:10:41.001585 31146 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:41.197304 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 31086
2024-05-01T20:10:41Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (106186 ms)
[----------] 1 test from RebalanceParamTest (106186 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240501 20:10:41.338683 29217 test_util.cc:268] Using random seed: 347102492
I20240501 20:10:41.340354 29217 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:10:41.340423 29217 ts_itest-base.cc:113] --------------
I20240501 20:10:41.340485 29217 ts_itest-base.cc:114] 8 tablet servers
I20240501 20:10:41.340576 29217 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:10:41.340636 29217 ts_itest-base.cc:116] --------------
2024-05-01T20:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:10:41Z Disabled control of system clock
I20240501 20:10:41.405486 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:33949
--webserver_interface=127.28.136.126
--webserver_port=0
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.136.126:33949
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:10:41.747929 4001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:41.748271 4001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:41.748402 4001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:41.755925 4001 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:41.756072 4001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:10:41.756157 4001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:41.756237 4001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:10:41.756299 4001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:10:41.765611 4001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.136.126:33949
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:33949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.136.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:41.766999 4001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:41.768903 4001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:41.779712 4006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:41.780114 4008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:41.780480 4010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:41.780866 4001 server_base.cc:1034] running on GCE node
I20240501 20:10:41.781580 4001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:41.782950 4001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:41.784132 4001 hybrid_clock.cc:648] HybridClock initialized: now 1714594241784075 us; error 50 us; skew 500 ppm
I20240501 20:10:41.784472 4001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:41.786633 4001 webserver.cc:460] Webserver started at http://127.28.136.126:41627/ using document root <none> and password file <none>
I20240501 20:10:41.787359 4001 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:41.787468 4001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:41.787781 4001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:41.790362 4001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7618d84946ab4e0096d32aaa40ff607d"
format_stamp: "Formatted at 2024-05-01 20:10:41 on dist-test-slave-sx0n"
I20240501 20:10:41.791085 4001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7618d84946ab4e0096d32aaa40ff607d"
format_stamp: "Formatted at 2024-05-01 20:10:41 on dist-test-slave-sx0n"
I20240501 20:10:41.795882 4001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240501 20:10:41.799077 4016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:41.800288 4001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:41.800446 4001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7618d84946ab4e0096d32aaa40ff607d"
format_stamp: "Formatted at 2024-05-01 20:10:41 on dist-test-slave-sx0n"
I20240501 20:10:41.800622 4001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:41.829246 4001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:41.830138 4001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:41.830437 4001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:41.847895 4001 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.126:33949
I20240501 20:10:41.847915 4071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.126:33949 every 8 connection(s)
I20240501 20:10:41.849262 4001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:10:41.853019 4072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:41.860481 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 4001
I20240501 20:10:41.861356 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:10:41.866559 4072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d: Bootstrap starting.
I20240501 20:10:41.869794 4072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:41.870851 4072 log.cc:824] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:41.873656 4072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d: No bootstrap required, opened a new log
I20240501 20:10:41.880386 4072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7618d84946ab4e0096d32aaa40ff607d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 33949 } }
I20240501 20:10:41.880843 4072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:41.880950 4072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7618d84946ab4e0096d32aaa40ff607d, State: Initialized, Role: FOLLOWER
I20240501 20:10:41.881529 4072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7618d84946ab4e0096d32aaa40ff607d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 33949 } }
I20240501 20:10:41.881734 4072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:10:41.881850 4072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:10:41.881994 4072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:41.884033 4072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7618d84946ab4e0096d32aaa40ff607d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 33949 } }
I20240501 20:10:41.884539 4072 leader_election.cc:302] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7618d84946ab4e0096d32aaa40ff607d; no voters:
I20240501 20:10:41.885128 4072 leader_election.cc:288] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:10:41.885248 4078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:41.885824 4078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [term 1 LEADER]: Becoming Leader. State: Replica: 7618d84946ab4e0096d32aaa40ff607d, State: Running, Role: LEADER
I20240501 20:10:41.886343 4078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7618d84946ab4e0096d32aaa40ff607d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 33949 } }
I20240501 20:10:41.887043 4072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:10:41.888803 4080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7618d84946ab4e0096d32aaa40ff607d. Latest consensus state: current_term: 1 leader_uuid: "7618d84946ab4e0096d32aaa40ff607d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7618d84946ab4e0096d32aaa40ff607d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 33949 } } }
I20240501 20:10:41.889117 4080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [sys.catalog]: This master's current role is: LEADER
I20240501 20:10:41.889314 4079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7618d84946ab4e0096d32aaa40ff607d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7618d84946ab4e0096d32aaa40ff607d" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 33949 } } }
I20240501 20:10:41.889626 4079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d [sys.catalog]: This master's current role is: LEADER
I20240501 20:10:41.893343 4083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:10:41.898933 4083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:10:41.907263 4083 catalog_manager.cc:1347] Generated new cluster ID: 0429393f2848448eb942fc5579746399
I20240501 20:10:41.907358 4083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:10:41.951730 4083 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:10:41.953543 4083 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:10:41.964722 4083 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7618d84946ab4e0096d32aaa40ff607d: Generated new TSK 0
I20240501 20:10:41.965785 4083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:10:41.974355 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.65:0
--local_ip_for_outbound_sockets=127.28.136.65
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 20:10:42.319974 4098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:42.320281 4098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:42.320394 4098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:42.327800 4098 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:42.327929 4098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:42.328059 4098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.65
I20240501 20:10:42.337209 4098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.136.65
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.65
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:42.338574 4098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:42.340315 4098 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:42.352478 4104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:42.352669 4105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:42.353117 4107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:42.353688 4098 server_base.cc:1034] running on GCE node
I20240501 20:10:42.354318 4098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:42.355281 4098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:42.356484 4098 hybrid_clock.cc:648] HybridClock initialized: now 1714594242356432 us; error 48 us; skew 500 ppm
I20240501 20:10:42.356830 4098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:42.359026 4098 webserver.cc:460] Webserver started at http://127.28.136.65:36263/ using document root <none> and password file <none>
I20240501 20:10:42.359766 4098 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:42.359879 4098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:42.360200 4098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:42.362905 4098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "63c1e379865d42e78ea6b6b1d908ece7"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-sx0n"
I20240501 20:10:42.363716 4098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "63c1e379865d42e78ea6b6b1d908ece7"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-sx0n"
I20240501 20:10:42.369184 4098 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240501 20:10:42.372656 4113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:42.374331 4098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:10:42.374511 4098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "63c1e379865d42e78ea6b6b1d908ece7"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-sx0n"
I20240501 20:10:42.374750 4098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:42.390156 4098 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:42.390977 4098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:42.391228 4098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:42.392246 4098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:42.393831 4098 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:42.393919 4098 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:42.394003 4098 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:42.394057 4098 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:42.422986 4098 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.65:43231
I20240501 20:10:42.423038 4230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.65:43231 every 8 connection(s)
I20240501 20:10:42.424332 4098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:10:42.430378 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 4098
I20240501 20:10:42.430575 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:10:42.433894 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.66:0
--local_ip_for_outbound_sockets=127.28.136.66
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:10:42.436997 4231 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:10:42.437306 4231 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:42.438120 4231 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:10:42.450366 4035 ts_manager.cc:194] Registered new tserver with Master: 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:10:42.452558 4035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.65:39183
W20240501 20:10:42.779817 4235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:42.780117 4235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:42.780233 4235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:42.787961 4235 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:42.788100 4235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:42.788275 4235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.66
I20240501 20:10:42.797593 4235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.136.66
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.66
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:42.799000 4235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:42.800802 4235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:42.812863 4240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:42.813153 4243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:42.813601 4241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:42.814111 4235 server_base.cc:1034] running on GCE node
I20240501 20:10:42.814699 4235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:42.815639 4235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:42.816813 4235 hybrid_clock.cc:648] HybridClock initialized: now 1714594242816759 us; error 45 us; skew 500 ppm
I20240501 20:10:42.817116 4235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:42.819301 4235 webserver.cc:460] Webserver started at http://127.28.136.66:42233/ using document root <none> and password file <none>
I20240501 20:10:42.820029 4235 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:42.820142 4235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:42.820472 4235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:42.823133 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4acbc8659ce24119bd77571d1171ee1d"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-sx0n"
I20240501 20:10:42.823892 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4acbc8659ce24119bd77571d1171ee1d"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-sx0n"
I20240501 20:10:42.828990 4235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240501 20:10:42.832290 4249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:42.833575 4235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:42.833724 4235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4acbc8659ce24119bd77571d1171ee1d"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-sx0n"
I20240501 20:10:42.833912 4235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:42.849867 4235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:42.850963 4235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:42.851267 4235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:42.852409 4235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:42.854302 4235 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:42.854404 4235 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:42.854504 4235 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:42.854578 4235 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:42.884820 4235 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.66:38397
I20240501 20:10:42.884891 4362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.66:38397 every 8 connection(s)
I20240501 20:10:42.886150 4235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:10:42.892818 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 4235
I20240501 20:10:42.893033 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:10:42.896795 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:0
--local_ip_for_outbound_sockets=127.28.136.67
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:10:42.900048 4363 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:10:42.900393 4363 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:42.901432 4363 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:10:42.902931 4035 ts_manager.cc:194] Registered new tserver with Master: 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:42.903735 4035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.66:49327
W20240501 20:10:43.234432 4366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:43.234730 4366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:43.234845 4366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:43.242323 4366 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:43.242448 4366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:43.242619 4366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:10:43.251534 4366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:43.252911 4366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:43.254602 4366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:43.266393 4371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:43.266808 4372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:43.267127 4374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:43.267644 4366 server_base.cc:1034] running on GCE node
I20240501 20:10:43.268225 4366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:43.269214 4366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:43.270404 4366 hybrid_clock.cc:648] HybridClock initialized: now 1714594243270346 us; error 46 us; skew 500 ppm
I20240501 20:10:43.270725 4366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:43.272914 4366 webserver.cc:460] Webserver started at http://127.28.136.67:45047/ using document root <none> and password file <none>
I20240501 20:10:43.273643 4366 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:43.273762 4366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:43.274067 4366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:43.276651 4366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "596d3e24651b4787916397f573c09f1d"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-sx0n"
I20240501 20:10:43.277442 4366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "596d3e24651b4787916397f573c09f1d"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-sx0n"
I20240501 20:10:43.282591 4366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:10:43.285877 4380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.287174 4366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:10:43.287322 4366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "596d3e24651b4787916397f573c09f1d"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-sx0n"
I20240501 20:10:43.287508 4366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:43.305761 4366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:43.306586 4366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:43.306875 4366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:43.307791 4366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:43.309250 4366 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:43.309345 4366 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.309448 4366 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:43.309515 4366 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.348474 4366 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:38865
I20240501 20:10:43.348526 4493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:38865 every 8 connection(s)
I20240501 20:10:43.349828 4366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:10:43.352223 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 4366
I20240501 20:10:43.352409 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:10:43.355793 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.68:0
--local_ip_for_outbound_sockets=127.28.136.68
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:10:43.362891 4494 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:10:43.363188 4494 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:43.364019 4494 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:10:43.365471 4035 ts_manager.cc:194] Registered new tserver with Master: 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:43.366272 4035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:60731
I20240501 20:10:43.455896 4231 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
W20240501 20:10:43.696594 4497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:43.696967 4497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:43.697121 4497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:43.704825 4497 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:43.704962 4497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:43.705140 4497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.68
I20240501 20:10:43.714443 4497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.136.68
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.68
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:43.715718 4497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:43.717460 4497 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:43.729449 4502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:43.729686 4505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:43.730118 4503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:43.730571 4497 server_base.cc:1034] running on GCE node
I20240501 20:10:43.731238 4497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:43.732208 4497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:43.733404 4497 hybrid_clock.cc:648] HybridClock initialized: now 1714594243733347 us; error 44 us; skew 500 ppm
I20240501 20:10:43.733726 4497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:43.740914 4497 webserver.cc:460] Webserver started at http://127.28.136.68:37431/ using document root <none> and password file <none>
I20240501 20:10:43.741732 4497 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:43.741855 4497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:43.742188 4497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:43.744958 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "75b5acf03b9f4f6e98a67ae046587e79"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-sx0n"
I20240501 20:10:43.745755 4497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "75b5acf03b9f4f6e98a67ae046587e79"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-sx0n"
I20240501 20:10:43.751300 4497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:10:43.755021 4511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.756430 4497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:43.756603 4497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "75b5acf03b9f4f6e98a67ae046587e79"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-sx0n"
I20240501 20:10:43.756858 4497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:43.771801 4497 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:43.772683 4497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:43.772974 4497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:43.773883 4497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:43.775346 4497 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:43.775439 4497 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.775545 4497 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:43.775614 4497 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.804473 4497 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.68:36125
I20240501 20:10:43.804519 4624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.68:36125 every 8 connection(s)
I20240501 20:10:43.805799 4497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:10:43.812749 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 4497
I20240501 20:10:43.812944 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:10:43.816116 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.69:0
--local_ip_for_outbound_sockets=127.28.136.69
--webserver_interface=127.28.136.69
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:10:43.818078 4625 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:10:43.818356 4625 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:43.819132 4625 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:10:43.820405 4035 ts_manager.cc:194] Registered new tserver with Master: 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:43.821218 4035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.68:49203
I20240501 20:10:43.906333 4363 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
W20240501 20:10:44.154768 4628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:44.155081 4628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:44.155227 4628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:44.162845 4628 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:44.162983 4628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:44.163151 4628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.69
I20240501 20:10:44.172318 4628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.136.69
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.69
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:44.173691 4628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:44.175447 4628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:44.187868 4633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:44.188277 4634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:44.188712 4636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:44.189137 4628 server_base.cc:1034] running on GCE node
I20240501 20:10:44.189838 4628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:44.190858 4628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:44.192093 4628 hybrid_clock.cc:648] HybridClock initialized: now 1714594244192031 us; error 50 us; skew 500 ppm
I20240501 20:10:44.192467 4628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:44.194833 4628 webserver.cc:460] Webserver started at http://127.28.136.69:39689/ using document root <none> and password file <none>
I20240501 20:10:44.195567 4628 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:44.195665 4628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:44.195963 4628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:44.198590 4628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ee4daecd449e4fc981fd1fe2b0e492c2"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-sx0n"
I20240501 20:10:44.199383 4628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ee4daecd449e4fc981fd1fe2b0e492c2"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-sx0n"
I20240501 20:10:44.204767 4628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 20:10:44.208226 4642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.209602 4628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:44.209746 4628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ee4daecd449e4fc981fd1fe2b0e492c2"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-sx0n"
I20240501 20:10:44.209921 4628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:44.234292 4628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:44.235193 4628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:44.235450 4628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:44.236423 4628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:44.238082 4628 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:44.238199 4628 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.238356 4628 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:44.238422 4628 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.268579 4628 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.69:45155
I20240501 20:10:44.268631 4758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.69:45155 every 8 connection(s)
I20240501 20:10:44.271039 4628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:10:44.281005 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 4628
I20240501 20:10:44.281229 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:10:44.283507 4759 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:10:44.283838 4759 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:44.284919 4759 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:10:44.284935 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.70:0
--local_ip_for_outbound_sockets=127.28.136.70
--webserver_interface=127.28.136.70
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:10:44.286547 4035 ts_manager.cc:194] Registered new tserver with Master: ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155)
I20240501 20:10:44.287596 4035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.69:37817
I20240501 20:10:44.368829 4494 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
W20240501 20:10:44.636233 4762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:44.636529 4762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:44.636662 4762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:44.644320 4762 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:44.644449 4762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:44.644642 4762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.70
I20240501 20:10:44.653745 4762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.136.70
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.70
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:44.655051 4762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:44.656791 4762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:44.668500 4767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:44.669004 4768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:44.669327 4770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:44.669790 4762 server_base.cc:1034] running on GCE node
I20240501 20:10:44.670357 4762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:44.671283 4762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:44.672495 4762 hybrid_clock.cc:648] HybridClock initialized: now 1714594244672447 us; error 41 us; skew 500 ppm
I20240501 20:10:44.672854 4762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:44.675038 4762 webserver.cc:460] Webserver started at http://127.28.136.70:37389/ using document root <none> and password file <none>
I20240501 20:10:44.675781 4762 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:44.675889 4762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:44.676218 4762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:44.678865 4762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "f5195730e32a4abd8519baf9961d32f2"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-sx0n"
I20240501 20:10:44.679620 4762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "f5195730e32a4abd8519baf9961d32f2"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-sx0n"
I20240501 20:10:44.684639 4762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 20:10:44.687861 4776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.689160 4762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:44.689311 4762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f5195730e32a4abd8519baf9961d32f2"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-sx0n"
I20240501 20:10:44.689491 4762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:44.707320 4762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:44.708175 4762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:44.708442 4762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:44.709376 4762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:44.710850 4762 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:44.710947 4762 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.711055 4762 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:44.711122 4762 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.740521 4762 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.70:40983
I20240501 20:10:44.740568 4892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.70:40983 every 8 connection(s)
I20240501 20:10:44.741858 4762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:10:44.742925 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 4762
I20240501 20:10:44.743113 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:10:44.747848 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.71:0
--local_ip_for_outbound_sockets=127.28.136.71
--webserver_interface=127.28.136.71
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:10:44.755270 4893 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:10:44.755566 4893 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:44.756393 4893 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:10:44.757848 4035 ts_manager.cc:194] Registered new tserver with Master: f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983)
I20240501 20:10:44.758651 4035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.70:41399
I20240501 20:10:44.823521 4625 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
W20240501 20:10:45.088971 4897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:45.089286 4897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:45.089426 4897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:45.097129 4897 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:45.097277 4897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:45.097445 4897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.71
I20240501 20:10:45.106495 4897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.136.71
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.71
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:45.107844 4897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:45.109602 4897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:45.121435 4903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:45.121870 4902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:45.122323 4905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:45.122675 4897 server_base.cc:1034] running on GCE node
I20240501 20:10:45.123246 4897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:45.124114 4897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:45.125299 4897 hybrid_clock.cc:648] HybridClock initialized: now 1714594245125243 us; error 49 us; skew 500 ppm
I20240501 20:10:45.125602 4897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:45.127830 4897 webserver.cc:460] Webserver started at http://127.28.136.71:33559/ using document root <none> and password file <none>
I20240501 20:10:45.128540 4897 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:45.128646 4897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:45.128979 4897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:45.131511 4897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "9592068f16ca406eb8a347a65d365782"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-sx0n"
I20240501 20:10:45.132251 4897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "9592068f16ca406eb8a347a65d365782"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-sx0n"
I20240501 20:10:45.137203 4897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240501 20:10:45.140544 4911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.141903 4897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:10:45.142050 4897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9592068f16ca406eb8a347a65d365782"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-sx0n"
I20240501 20:10:45.142261 4897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:45.175767 4897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:45.176651 4897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:45.176949 4897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:45.177923 4897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:45.179396 4897 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:45.179487 4897 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.179596 4897 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:45.179664 4897 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.208829 4897 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.71:40971
I20240501 20:10:45.208896 5023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.71:40971 every 8 connection(s)
I20240501 20:10:45.210165 4897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:10:45.214300 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 4897
I20240501 20:10:45.214495 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:10:45.217530 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.72:0
--local_ip_for_outbound_sockets=127.28.136.72
--webserver_interface=127.28.136.72
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:10:45.222826 5024 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:10:45.223100 5024 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:45.223903 5024 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:10:45.225284 4035 ts_manager.cc:194] Registered new tserver with Master: 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971)
I20240501 20:10:45.226241 4035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.71:33581
I20240501 20:10:45.290550 4759 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
W20240501 20:10:45.560992 5028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:45.561372 5028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:45.561525 5028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:45.569557 5028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:45.569702 5028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:45.569864 5028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.72
I20240501 20:10:45.578936 5028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.136.72
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.72
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:45.580410 5028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:45.582247 5028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:45.594234 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:45.594707 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:45.595044 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:45.595461 5028 server_base.cc:1034] running on GCE node
I20240501 20:10:45.596087 5028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:45.597115 5028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:45.598340 5028 hybrid_clock.cc:648] HybridClock initialized: now 1714594245598289 us; error 51 us; skew 500 ppm
I20240501 20:10:45.598670 5028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:45.601017 5028 webserver.cc:460] Webserver started at http://127.28.136.72:34873/ using document root <none> and password file <none>
I20240501 20:10:45.601919 5028 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:45.602027 5028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:45.602349 5028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:45.605212 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "55fe2a4464a9402f927b2b60609b34dd"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-sx0n"
I20240501 20:10:45.606009 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "55fe2a4464a9402f927b2b60609b34dd"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-sx0n"
I20240501 20:10:45.611092 5028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:10:45.614559 5043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.615942 5028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:45.616096 5028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "55fe2a4464a9402f927b2b60609b34dd"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-sx0n"
I20240501 20:10:45.616291 5028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:45.638631 5028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:45.639621 5028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:45.639912 5028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:45.640959 5028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:45.642604 5028 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:45.642705 5028 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.642807 5028 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:45.642877 5028 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.672649 5028 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.72:46003
I20240501 20:10:45.672700 5156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.72:46003 every 8 connection(s)
I20240501 20:10:45.674008 5028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:10:45.674505 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 5028
I20240501 20:10:45.674681 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:10:45.686676 5157 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:10:45.686995 5157 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:45.687945 5157 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:10:45.689486 4035 ts_manager.cc:194] Registered new tserver with Master: 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003)
I20240501 20:10:45.690414 4035 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.72:37561
I20240501 20:10:45.691291 29217 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:10:45.708012 29217 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:10:45.708140 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 4628
I20240501 20:10:45.715806 4754 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:45.761154 4893 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
I20240501 20:10:45.801129 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 4628
I20240501 20:10:45.815340 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 4762
I20240501 20:10:45.823172 4888 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:45.911397 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 4762
I20240501 20:10:45.925130 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 4897
I20240501 20:10:45.932924 5019 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:46.011605 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 4897
I20240501 20:10:46.025518 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 5028
I20240501 20:10:46.033558 5152 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:46.113873 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 5028
I20240501 20:10:49.879357 4033 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:49.915314 4428 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.916970 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.917426 4429 tablet_service.cc:1467] Processing CreateTablet for tablet d09b0474ecbf4d458d7a520c9e9f329b (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.918426 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09b0474ecbf4d458d7a520c9e9f329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.926249 4559 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.926296 4298 tablet_service.cc:1467] Processing CreateTablet for tablet d09b0474ecbf4d458d7a520c9e9f329b (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.927405 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.927618 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09b0474ecbf4d458d7a520c9e9f329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.929279 4560 tablet_service.cc:1467] Processing CreateTablet for tablet b344760b82cb4bc09379d990f9558121 (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.929811 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b344760b82cb4bc09379d990f9558121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.930792 4297 tablet_service.cc:1467] Processing CreateTablet for tablet b344760b82cb4bc09379d990f9558121 (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.931242 4297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b344760b82cb4bc09379d990f9558121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.936805 4164 tablet_service.cc:1467] Processing CreateTablet for tablet b344760b82cb4bc09379d990f9558121 (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.937671 4163 tablet_service.cc:1467] Processing CreateTablet for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.938891 4163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.937881 4165 tablet_service.cc:1467] Processing CreateTablet for tablet d09b0474ecbf4d458d7a520c9e9f329b (DEFAULT_TABLE table=rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:49.941171 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09b0474ecbf4d458d7a520c9e9f329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.953116 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b344760b82cb4bc09379d990f9558121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:49.960427 5186 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:49.960427 5188 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:49.964088 5188 tablet_bootstrap.cc:654] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:49.964088 5186 tablet_bootstrap.cc:654] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:49.965430 5188 log.cc:824] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:49.965639 5186 log.cc:824] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:49.967032 5187 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:49.969684 5189 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:49.970922 5187 tablet_bootstrap.cc:654] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:49.972229 5189 tablet_bootstrap.cc:654] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:49.972280 5187 log.cc:824] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:49.973438 5189 log.cc:824] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:49.975534 5188 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:49.975903 5188 ts_tablet_manager.cc:1367] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240501 20:10:49.978339 5187 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:49.978420 5189 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:49.978717 5189 ts_tablet_manager.cc:1367] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:10:49.978775 5187 ts_tablet_manager.cc:1367] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20240501 20:10:49.982709 5186 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:49.983023 5186 ts_tablet_manager.cc:1367] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.013s
I20240501 20:10:49.985376 5188 raft_consensus.cc:348] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:49.985857 5188 raft_consensus.cc:374] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:49.985991 5188 raft_consensus.cc:729] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:49.986721 5188 consensus_queue.cc:260] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:49.987237 5189 raft_consensus.cc:348] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:49.987669 5189 raft_consensus.cc:374] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:49.987797 5189 raft_consensus.cc:729] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:49.987556 5187 raft_consensus.cc:348] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:49.988135 5187 raft_consensus.cc:374] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:49.988284 5187 raft_consensus.cc:729] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:49.988494 5189 consensus_queue.cc:260] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:49.989049 5187 consensus_queue.cc:260] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:49.992061 5186 raft_consensus.cc:348] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:49.992516 5186 raft_consensus.cc:374] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:49.992660 5186 raft_consensus.cc:729] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:49.993489 5187 ts_tablet_manager.cc:1398] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20240501 20:10:49.993366 5186 consensus_queue.cc:260] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.001173 5187 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:50.003131 5187 tablet_bootstrap.cc:654] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.005769 5188 ts_tablet_manager.cc:1398] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.030s user 0.010s sys 0.017s
I20240501 20:10:50.006176 5189 ts_tablet_manager.cc:1398] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.027s user 0.010s sys 0.015s
I20240501 20:10:50.006275 5188 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:50.006651 5189 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:50.007303 5187 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:50.007504 5187 ts_tablet_manager.cc:1367] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:10:50.007925 5188 tablet_bootstrap.cc:654] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.008334 5189 tablet_bootstrap.cc:654] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.015766 5187 raft_consensus.cc:348] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.016070 5187 raft_consensus.cc:374] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.016212 5187 raft_consensus.cc:729] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.016559 5187 consensus_queue.cc:260] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.016863 5188 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:50.017050 5188 ts_tablet_manager.cc:1367] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 20:10:50.017992 5189 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:50.018226 5189 ts_tablet_manager.cc:1367] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240501 20:10:50.018126 5188 raft_consensus.cc:348] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:50.018424 5188 raft_consensus.cc:374] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.018571 5188 raft_consensus.cc:729] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.018893 5188 consensus_queue.cc:260] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:50.019356 5187 ts_tablet_manager.cc:1398] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 20:10:50.019249 5189 raft_consensus.cc:348] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.019434 5188 ts_tablet_manager.cc:1398] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:50.019456 5189 raft_consensus.cc:374] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.019609 5189 raft_consensus.cc:729] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.019917 5189 consensus_queue.cc:260] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.020438 5189 ts_tablet_manager.cc:1398] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:50.020759 5189 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:50.021180 5186 ts_tablet_manager.cc:1398] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.038s user 0.012s sys 0.022s
I20240501 20:10:50.021648 5186 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:50.022341 5189 tablet_bootstrap.cc:654] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.023232 5186 tablet_bootstrap.cc:654] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.026796 5189 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:50.026958 5189 ts_tablet_manager.cc:1367] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240501 20:10:50.027611 5189 raft_consensus.cc:348] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.027765 5186 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:50.027848 5189 raft_consensus.cc:374] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.027895 5186 ts_tablet_manager.cc:1367] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:10:50.027925 5189 raft_consensus.cc:729] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.028169 5189 consensus_queue.cc:260] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.028717 5189 ts_tablet_manager.cc:1398] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:50.028890 5186 raft_consensus.cc:348] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.029054 5186 raft_consensus.cc:374] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.029132 5186 raft_consensus.cc:729] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.029345 5186 consensus_queue.cc:260] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.029785 5186 ts_tablet_manager.cc:1398] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:50.049357 5195 raft_consensus.cc:482] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:50.049604 5195 raft_consensus.cc:504] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.051304 5195 leader_election.cc:288] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:50.058494 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:10:50.058498 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:50.058956 4318 raft_consensus.cc:2454] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:50.058957 4449 raft_consensus.cc:2454] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:50.059547 4118 leader_election.cc:302] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:50.059937 5195 raft_consensus.cc:2781] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:50.060091 5195 raft_consensus.cc:482] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:50.060179 5195 raft_consensus.cc:3037] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:10:50.061770 4626 tablet.cc:2369] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:50.062671 5195 raft_consensus.cc:504] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.063305 5195 leader_election.cc:288] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:50.063711 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:50.063843 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:10:50.063932 4449 raft_consensus.cc:3037] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:50.064050 4318 raft_consensus.cc:3037] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:50.067030 4318 raft_consensus.cc:2454] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:50.067031 4449 raft_consensus.cc:2454] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:50.067420 4118 leader_election.cc:302] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:50.067750 5195 raft_consensus.cc:2781] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:50.068145 5195 raft_consensus.cc:686] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:50.068574 5195 consensus_queue.cc:237] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.073246 4032 catalog_manager.cc:5484] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:50.106137 4495 tablet.cc:2369] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:10:50.142462 4364 tablet.cc:2369] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:10:50.181890 4232 tablet.cc:2369] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:50.296365 5194 raft_consensus.cc:482] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:50.296613 5194 raft_consensus.cc:504] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:50.298326 5194 leader_election.cc:288] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:10:50.305445 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b344760b82cb4bc09379d990f9558121" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:50.305445 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b344760b82cb4bc09379d990f9558121" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:10:50.305905 4186 raft_consensus.cc:2454] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:50.305984 4580 raft_consensus.cc:2454] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:50.306488 4251 leader_election.cc:302] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:50.306874 5194 raft_consensus.cc:2781] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:50.306986 5194 raft_consensus.cc:482] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:50.307073 5194 raft_consensus.cc:3037] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:50.309087 5194 raft_consensus.cc:504] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:50.309713 5194 leader_election.cc:288] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:10:50.310155 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b344760b82cb4bc09379d990f9558121" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:50.310304 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b344760b82cb4bc09379d990f9558121" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:10:50.310364 4580 raft_consensus.cc:3037] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:50.310504 4186 raft_consensus.cc:3037] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:50.313422 4186 raft_consensus.cc:2454] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:50.313422 4580 raft_consensus.cc:2454] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:50.314071 4253 leader_election.cc:302] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:50.314381 5194 raft_consensus.cc:2781] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:50.314774 5194 raft_consensus.cc:686] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:10:50.315299 5194 consensus_queue.cc:237] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:50.320107 4032 catalog_manager.cc:5484] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:50.323184 5196 raft_consensus.cc:482] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:50.323347 5196 raft_consensus.cc:504] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.324956 5196 leader_election.cc:288] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:10:50.330619 5195 raft_consensus.cc:482] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:50.330829 5195 raft_consensus.cc:504] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.331648 5195 leader_election.cc:288] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:50.332679 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:50.332994 4449 raft_consensus.cc:2454] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:50.333782 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:50.334005 4449 raft_consensus.cc:2454] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:10:50.334414 4115 leader_election.cc:302] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:50.334726 5195 raft_consensus.cc:2781] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:50.334825 5195 raft_consensus.cc:482] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:50.334915 5195 raft_consensus.cc:3037] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:50.335026 4513 leader_election.cc:302] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:50.335482 5196 raft_consensus.cc:2781] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:50.335602 5196 raft_consensus.cc:482] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:50.335683 5196 raft_consensus.cc:3037] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:50.337663 5195 raft_consensus.cc:504] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.338459 5195 leader_election.cc:288] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:50.338438 5196 raft_consensus.cc:504] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.338928 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:50.339181 4449 raft_consensus.cc:3037] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:50.339330 5196 leader_election.cc:288] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:10:50.341172 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:50.342413 4185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:10:50.342805 4185 raft_consensus.cc:2379] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 75b5acf03b9f4f6e98a67ae046587e79 in current term 1: Already voted for candidate 63c1e379865d42e78ea6b6b1d908ece7 in this term.
I20240501 20:10:50.343515 4513 leader_election.cc:302] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75b5acf03b9f4f6e98a67ae046587e79; no voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7
I20240501 20:10:50.343847 5196 raft_consensus.cc:2726] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:10:50.341174 4449 raft_consensus.cc:2454] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:50.344245 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:10:50.344354 4115 leader_election.cc:302] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:50.344513 4186 raft_consensus.cc:2379] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75b5acf03b9f4f6e98a67ae046587e79 in current term 1: Already voted for candidate 63c1e379865d42e78ea6b6b1d908ece7 in this term.
I20240501 20:10:50.344627 5195 raft_consensus.cc:2781] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:50.344899 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:50.344872 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:50.345142 4580 raft_consensus.cc:2379] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 63c1e379865d42e78ea6b6b1d908ece7 in current term 1: Already voted for candidate 75b5acf03b9f4f6e98a67ae046587e79 in this term.
I20240501 20:10:50.345439 5195 raft_consensus.cc:686] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:50.345747 5195 consensus_queue.cc:237] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:50.348385 4032 catalog_manager.cc:5484] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:50.599422 5195 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:50.605551 5217 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:50.704102 5214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:50.704442 5214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:50.711639 5214 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 20:10:50.740644 5217 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:50.746917 5217 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:50.879716 4580 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:50.880630 5194 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:50.888082 4186 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:50.889719 5194 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:50.968561 5261 mvcc.cc:205] Tried to move back new op lower bound from 7022978051574419456 to 7022978049295278080. Current Snapshot: MvccSnapshot[applied={T|T < 7022978051574419456}]
I20240501 20:10:51.012907 5216 mvcc.cc:205] Tried to move back new op lower bound from 7022978051574419456 to 7022978049295278080. Current Snapshot: MvccSnapshot[applied={T|T < 7022978051574419456 or (T in {7022978051574419456})}]
INSERT report
rows total: 2000
time total: 310.621 ms
time per row: 0.155311 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:10:52.031059 4032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:52.040620 4298 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdac8c0b92d442ebd84fb43af7e32fb (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.041208 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdac8c0b92d442ebd84fb43af7e32fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.041277 4559 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdac8c0b92d442ebd84fb43af7e32fb (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.041699 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdac8c0b92d442ebd84fb43af7e32fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.042874 4428 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdac8c0b92d442ebd84fb43af7e32fb (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.043285 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdac8c0b92d442ebd84fb43af7e32fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.047766 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 2e97fe7ab97a4458b5e977af3d1356df (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.048115 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e97fe7ab97a4458b5e977af3d1356df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.049077 4429 tablet_service.cc:1467] Processing CreateTablet for tablet 2e97fe7ab97a4458b5e977af3d1356df (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.049443 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e97fe7ab97a4458b5e977af3d1356df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.053112 4427 tablet_service.cc:1467] Processing CreateTablet for tablet 0a452749c5c44b87bb495983853cea3a (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.053454 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a452749c5c44b87bb495983853cea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.057533 4164 tablet_service.cc:1467] Processing CreateTablet for tablet 0a452749c5c44b87bb495983853cea3a (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.057976 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a452749c5c44b87bb495983853cea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.062842 4297 tablet_service.cc:1467] Processing CreateTablet for tablet 2e97fe7ab97a4458b5e977af3d1356df (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.063311 4297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e97fe7ab97a4458b5e977af3d1356df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.069147 4296 tablet_service.cc:1467] Processing CreateTablet for tablet 0a452749c5c44b87bb495983853cea3a (DEFAULT_TABLE table=rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.069559 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a452749c5c44b87bb495983853cea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.076911 5284 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:52.077371 5285 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:52.078640 5282 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:52.079046 5284 tablet_bootstrap.cc:654] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.080462 5282 tablet_bootstrap.cc:654] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.081707 5283 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:52.082798 5285 tablet_bootstrap.cc:654] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.084040 5283 tablet_bootstrap.cc:654] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.084651 5285 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:52.084821 5285 ts_tablet_manager.cc:1367] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:10:52.085583 5285 raft_consensus.cc:348] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.085758 5282 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:52.085821 5285 raft_consensus.cc:374] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.085899 5282 ts_tablet_manager.cc:1367] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:10:52.085942 5285 raft_consensus.cc:729] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.086228 5285 consensus_queue.cc:260] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.086650 5282 raft_consensus.cc:348] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.086891 5282 raft_consensus.cc:374] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.087045 5282 raft_consensus.cc:729] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.087371 5282 consensus_queue.cc:260] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.088021 5285 ts_tablet_manager.cc:1398] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:52.088342 5285 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:52.089936 5283 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:52.090157 5283 ts_tablet_manager.cc:1367] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:10:52.090955 5284 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:52.091069 5283 raft_consensus.cc:348] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.091122 5284 ts_tablet_manager.cc:1367] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240501 20:10:52.091393 5283 raft_consensus.cc:374] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.091508 5283 raft_consensus.cc:729] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.091746 5283 consensus_queue.cc:260] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.092294 5284 raft_consensus.cc:348] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.092468 5283 ts_tablet_manager.cc:1398] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:52.092574 5284 raft_consensus.cc:374] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.092726 5284 raft_consensus.cc:729] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.092826 5283 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:52.093250 5284 consensus_queue.cc:260] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.093973 5284 ts_tablet_manager.cc:1398] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:52.094447 5283 tablet_bootstrap.cc:654] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.095348 5285 tablet_bootstrap.cc:654] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.097801 5282 ts_tablet_manager.cc:1398] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240501 20:10:52.098181 5282 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:52.100625 5285 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:52.100826 5285 ts_tablet_manager.cc:1367] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 20:10:52.101578 5285 raft_consensus.cc:348] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.101816 5285 raft_consensus.cc:374] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.101891 5285 raft_consensus.cc:729] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.102137 5285 consensus_queue.cc:260] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.102701 5285 ts_tablet_manager.cc:1398] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:52.103739 5282 tablet_bootstrap.cc:654] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.104158 5285 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:52.105705 5285 tablet_bootstrap.cc:654] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.110380 5283 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:52.110656 5283 ts_tablet_manager.cc:1367] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240501 20:10:52.111606 5283 raft_consensus.cc:348] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.111881 5283 raft_consensus.cc:374] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.111963 5283 raft_consensus.cc:729] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.112250 5283 consensus_queue.cc:260] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.112519 5217 raft_consensus.cc:482] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:52.112716 5217 raft_consensus.cc:504] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.113426 5282 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:52.113472 5217 leader_election.cc:288] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:52.113598 5282 ts_tablet_manager.cc:1367] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240501 20:10:52.114435 5282 raft_consensus.cc:348] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.114475 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a452749c5c44b87bb495983853cea3a" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:52.114660 5282 raft_consensus.cc:374] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.114706 4449 raft_consensus.cc:2454] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:52.114725 5282 raft_consensus.cc:729] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.115063 5282 consensus_queue.cc:260] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.115154 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a452749c5c44b87bb495983853cea3a" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:10:52.115373 4318 raft_consensus.cc:2454] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:52.115489 5283 ts_tablet_manager.cc:1398] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:52.115566 5282 ts_tablet_manager.cc:1398] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:52.116315 4118 leader_election.cc:302] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:52.117297 5217 raft_consensus.cc:2781] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:52.117403 5217 raft_consensus.cc:482] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:52.117501 5217 raft_consensus.cc:3037] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.117815 5283 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:52.120085 5217 raft_consensus.cc:504] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.120414 5283 tablet_bootstrap.cc:654] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.121451 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a452749c5c44b87bb495983853cea3a" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:10:52.121518 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a452749c5c44b87bb495983853cea3a" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:52.121681 4318 raft_consensus.cc:3037] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.121724 4449 raft_consensus.cc:3037] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.124524 4449 raft_consensus.cc:2454] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:52.125026 4115 leader_election.cc:302] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:52.124668 4318 raft_consensus.cc:2454] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:52.125739 5217 leader_election.cc:288] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:52.125998 5217 raft_consensus.cc:2781] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:52.126186 5217 raft_consensus.cc:686] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:52.126544 5217 consensus_queue.cc:237] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:52.128768 5285 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:52.129019 5285 ts_tablet_manager.cc:1367] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20240501 20:10:52.130061 5285 raft_consensus.cc:348] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.131281 5285 raft_consensus.cc:374] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.131544 5285 raft_consensus.cc:729] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.132098 5285 consensus_queue.cc:260] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.134646 5285 ts_tablet_manager.cc:1398] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 20:10:52.138962 4029 catalog_manager.cc:5484] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.139910 5283 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:52.140092 5283 ts_tablet_manager.cc:1367] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.013s
I20240501 20:10:52.140774 5283 raft_consensus.cc:348] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.140996 5283 raft_consensus.cc:374] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.141085 5283 raft_consensus.cc:729] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.141324 5283 consensus_queue.cc:260] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.143002 5283 ts_tablet_manager.cc:1398] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:52.153438 5262 raft_consensus.cc:482] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:52.153693 5262 raft_consensus.cc:504] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.154693 5262 leader_election.cc:288] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:52.155572 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:52.155889 4579 raft_consensus.cc:2454] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:52.156373 4253 leader_election.cc:302] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:52.156636 5262 raft_consensus.cc:2781] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:52.156801 5262 raft_consensus.cc:482] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:52.156906 5262 raft_consensus.cc:3037] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.158975 5262 raft_consensus.cc:504] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.159623 5262 leader_election.cc:288] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:52.160171 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:52.160395 4579 raft_consensus.cc:3037] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.162899 4579 raft_consensus.cc:2454] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:52.163436 4253 leader_election.cc:302] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:52.163856 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:52.163861 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:52.163986 5262 raft_consensus.cc:2781] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:52.164139 4449 raft_consensus.cc:2454] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:52.164641 5262 raft_consensus.cc:686] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:10:52.165143 5262 consensus_queue.cc:237] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:52.168190 4029 catalog_manager.cc:5484] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.333794 5262 raft_consensus.cc:482] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:52.334013 5262 raft_consensus.cc:504] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.334652 5262 leader_election.cc:288] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:52.335269 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e97fe7ab97a4458b5e977af3d1356df" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:52.335258 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e97fe7ab97a4458b5e977af3d1356df" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:52.335486 4579 raft_consensus.cc:2454] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:52.335500 4449 raft_consensus.cc:2454] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:52.335906 4251 leader_election.cc:302] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters:
I20240501 20:10:52.336206 5262 raft_consensus.cc:2781] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:52.336306 5262 raft_consensus.cc:482] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:52.336407 5262 raft_consensus.cc:3037] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.338395 5262 raft_consensus.cc:504] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.339028 5262 leader_election.cc:288] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:52.339502 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e97fe7ab97a4458b5e977af3d1356df" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:52.339514 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e97fe7ab97a4458b5e977af3d1356df" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:52.339713 4579 raft_consensus.cc:3037] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.339733 4449 raft_consensus.cc:3037] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.342492 4579 raft_consensus.cc:2454] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:52.342479 4449 raft_consensus.cc:2454] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:52.342962 4251 leader_election.cc:302] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters:
I20240501 20:10:52.343262 5262 raft_consensus.cc:2781] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:52.343427 5262 raft_consensus.cc:686] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:10:52.343705 5262 consensus_queue.cc:237] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:52.346496 4029 catalog_manager.cc:5484] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.578483 4449 raft_consensus.cc:3037] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.581362 5262 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.597865 5262 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.631745 5217 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.639696 5217 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:52.711562 5304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:52.711877 5304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:52.719153 5304 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:10:52.855530 4449 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:52.856221 4579 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:52.856962 5262 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.861135 5262 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.876994 5307 mvcc.cc:205] Tried to move back new op lower bound from 7022978059675475968 to 7022978057602121728. Current Snapshot: MvccSnapshot[applied={T|T < 7022978059675475968}]
I20240501 20:10:52.889554 5306 mvcc.cc:205] Tried to move back new op lower bound from 7022978059675475968 to 7022978057602121728. Current Snapshot: MvccSnapshot[applied={T|T < 7022978059675475968 or (T in {7022978059675475968})}]
INSERT report
rows total: 2000
time total: 238.324 ms
time per row: 0.119162 ms
I20240501 20:10:53.097388 4027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:53.106025 4559 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7fb3f245414d05a8e5d837076195c2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.106132 4298 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7fb3f245414d05a8e5d837076195c2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.106505 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7fb3f245414d05a8e5d837076195c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.106514 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7fb3f245414d05a8e5d837076195c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.107331 4296 tablet_service.cc:1467] Processing CreateTablet for tablet ca4d34ed427e44ed88379eb69feaeb8a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.107650 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4d34ed427e44ed88379eb69feaeb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.109769 4428 tablet_service.cc:1467] Processing CreateTablet for tablet 1564f07420a346dbaa77ef6a2a1a1b42 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.109850 4164 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7fb3f245414d05a8e5d837076195c2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.110168 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1564f07420a346dbaa77ef6a2a1a1b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.110270 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7fb3f245414d05a8e5d837076195c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.113131 4163 tablet_service.cc:1467] Processing CreateTablet for tablet ca4d34ed427e44ed88379eb69feaeb8a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.113484 4163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4d34ed427e44ed88379eb69feaeb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.113480 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 1564f07420a346dbaa77ef6a2a1a1b42 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.113579 4558 tablet_service.cc:1467] Processing CreateTablet for tablet ca4d34ed427e44ed88379eb69feaeb8a (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.113868 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1564f07420a346dbaa77ef6a2a1a1b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.113979 4165 tablet_service.cc:1467] Processing CreateTablet for tablet 1564f07420a346dbaa77ef6a2a1a1b42 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:53.114341 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1564f07420a346dbaa77ef6a2a1a1b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.120075 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4d34ed427e44ed88379eb69feaeb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:53.124424 5374 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:53.124563 5373 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:53.126808 5374 tablet_bootstrap.cc:654] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.126806 5373 tablet_bootstrap.cc:654] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.128468 5371 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:53.130446 5371 tablet_bootstrap.cc:654] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.130959 5372 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:53.133760 5372 tablet_bootstrap.cc:654] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.134220 5373 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:53.134383 5373 ts_tablet_manager.cc:1367] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 20:10:53.134666 5374 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:53.134858 5374 ts_tablet_manager.cc:1367] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240501 20:10:53.135170 5373 raft_consensus.cc:348] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.135351 5371 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:53.135466 5373 raft_consensus.cc:374] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.135511 5371 ts_tablet_manager.cc:1367] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:10:53.135605 5373 raft_consensus.cc:729] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.135686 5374 raft_consensus.cc:348] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.135958 5374 raft_consensus.cc:374] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.135941 5373 consensus_queue.cc:260] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.136091 5374 raft_consensus.cc:729] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.136322 5371 raft_consensus.cc:348] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.136370 5374 consensus_queue.cc:260] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.136575 5371 raft_consensus.cc:374] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.136689 5373 ts_tablet_manager.cc:1398] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:53.136916 5371 raft_consensus.cc:729] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.137042 5374 ts_tablet_manager.cc:1398] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:53.137188 5372 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:53.137362 5372 ts_tablet_manager.cc:1367] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:10:53.137239 5371 consensus_queue.cc:260] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.137475 5374 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:53.137995 5371 ts_tablet_manager.cc:1398] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:53.138367 5371 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:53.138247 5372 raft_consensus.cc:348] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.138445 5372 raft_consensus.cc:374] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.138576 5372 raft_consensus.cc:729] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.138875 5372 consensus_queue.cc:260] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.139423 5374 tablet_bootstrap.cc:654] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.139530 5372 ts_tablet_manager.cc:1398] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:53.141873 5373 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:53.142436 5374 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:53.142628 5374 ts_tablet_manager.cc:1367] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:10:53.142731 5371 tablet_bootstrap.cc:654] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.143405 5374 raft_consensus.cc:348] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.143675 5374 raft_consensus.cc:374] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.143729 5373 tablet_bootstrap.cc:654] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.143810 5374 raft_consensus.cc:729] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.144126 5374 consensus_queue.cc:260] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.144793 5374 ts_tablet_manager.cc:1398] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:53.145123 5374 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:53.146682 5374 tablet_bootstrap.cc:654] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.155951 5373 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:53.156119 5373 ts_tablet_manager.cc:1367] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20240501 20:10:53.156495 5371 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:53.156509 5374 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:53.156647 5374 ts_tablet_manager.cc:1367] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20240501 20:10:53.156668 5371 ts_tablet_manager.cc:1367] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.005s
I20240501 20:10:53.156803 5373 raft_consensus.cc:348] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.156994 5373 raft_consensus.cc:374] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.157068 5373 raft_consensus.cc:729] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.157310 5373 consensus_queue.cc:260] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.157421 5374 raft_consensus.cc:348] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.157656 5374 raft_consensus.cc:374] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.157421 5371 raft_consensus.cc:348] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.157775 5374 raft_consensus.cc:729] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.157810 5373 ts_tablet_manager.cc:1398] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:53.157814 5371 raft_consensus.cc:374] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.157891 5371 raft_consensus.cc:729] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.158025 5374 consensus_queue.cc:260] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.158186 5373 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:53.158171 5371 consensus_queue.cc:260] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.158574 5374 ts_tablet_manager.cc:1398] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:53.159166 5364 raft_consensus.cc:482] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:53.159322 5364 raft_consensus.cc:504] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.159817 5371 ts_tablet_manager.cc:1398] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:53.160006 5364 leader_election.cc:288] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:53.160835 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:53.161047 4579 raft_consensus.cc:2454] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:53.161465 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:10:53.161664 4318 raft_consensus.cc:2454] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:53.161993 5373 tablet_bootstrap.cc:654] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.162366 4118 leader_election.cc:302] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:53.162593 5364 raft_consensus.cc:2781] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:53.162703 5364 raft_consensus.cc:482] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:53.162797 5364 raft_consensus.cc:3037] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.165351 5364 raft_consensus.cc:504] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.165643 5373 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:53.165783 5373 ts_tablet_manager.cc:1367] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 20:10:53.165890 5364 leader_election.cc:288] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:53.166361 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:53.166476 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:10:53.166556 4579 raft_consensus.cc:3037] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.166625 4318 raft_consensus.cc:3037] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.166560 5373 raft_consensus.cc:348] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.166831 5373 raft_consensus.cc:374] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.166926 5373 raft_consensus.cc:729] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.167157 5373 consensus_queue.cc:260] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.167724 5373 ts_tablet_manager.cc:1398] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:53.169070 4579 raft_consensus.cc:2454] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:53.169081 4318 raft_consensus.cc:2454] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:53.169457 4118 leader_election.cc:302] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:53.169723 5364 raft_consensus.cc:2781] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:53.169875 5364 raft_consensus.cc:686] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:53.170130 5364 consensus_queue.cc:237] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:53.172663 4027 catalog_manager.cc:5484] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:53.209546 5364 raft_consensus.cc:482] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:53.209728 5364 raft_consensus.cc:504] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.210304 5364 leader_election.cc:288] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:53.210728 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1564f07420a346dbaa77ef6a2a1a1b42" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:53.210808 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1564f07420a346dbaa77ef6a2a1a1b42" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:53.210952 4579 raft_consensus.cc:2454] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:53.211025 4449 raft_consensus.cc:2454] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:53.211365 4118 leader_election.cc:302] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:53.211658 5364 raft_consensus.cc:2781] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:53.211757 5364 raft_consensus.cc:482] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:53.211861 5364 raft_consensus.cc:3037] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.213851 5364 raft_consensus.cc:504] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.214427 5364 leader_election.cc:288] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:53.214915 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1564f07420a346dbaa77ef6a2a1a1b42" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:53.215106 4449 raft_consensus.cc:3037] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.215214 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1564f07420a346dbaa77ef6a2a1a1b42" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:53.215373 4579 raft_consensus.cc:3037] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.216962 4449 raft_consensus.cc:2454] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:53.217262 4579 raft_consensus.cc:2454] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:53.217310 4115 leader_election.cc:302] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:53.217578 5364 raft_consensus.cc:2781] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:53.217752 5364 raft_consensus.cc:686] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:53.218006 5364 consensus_queue.cc:237] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:53.220593 4027 catalog_manager.cc:5484] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:53.327172 5364 raft_consensus.cc:482] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:53.327389 5364 raft_consensus.cc:504] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.328034 5364 leader_election.cc:288] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:53.328670 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:53.328713 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:10:53.328922 4579 raft_consensus.cc:2454] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:53.328997 4318 raft_consensus.cc:2454] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:53.329344 4118 leader_election.cc:302] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:53.329600 5364 raft_consensus.cc:2781] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:53.329702 5364 raft_consensus.cc:482] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:53.329802 5364 raft_consensus.cc:3037] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.331637 5364 raft_consensus.cc:504] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.332178 5364 leader_election.cc:288] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:53.332653 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:10:53.332653 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:53.332885 4579 raft_consensus.cc:3037] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.332901 4318 raft_consensus.cc:3037] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.335688 4579 raft_consensus.cc:2454] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:53.335676 4318 raft_consensus.cc:2454] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:53.336071 4118 leader_election.cc:302] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:53.336326 5364 raft_consensus.cc:2781] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:53.336454 5364 raft_consensus.cc:686] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:53.336709 5364 consensus_queue.cc:237] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:53.339510 4027 catalog_manager.cc:5484] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:53.614346 5364 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:53.620575 5390 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:53.690789 5391 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:53.696619 5391 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:53.715517 5387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:53.715864 5387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:53.724085 5387 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 20:10:53.873106 4317 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:53.873749 4578 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:53.877285 5217 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:53.878187 5217 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 235.741 ms
time per row: 0.117871 ms
I20240501 20:10:54.099189 4028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:54.108161 4428 tablet_service.cc:1467] Processing CreateTablet for tablet 126f7b0e88774d51ac087e5a1f758c78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.108480 4296 tablet_service.cc:1467] Processing CreateTablet for tablet 126f7b0e88774d51ac087e5a1f758c78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.108654 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126f7b0e88774d51ac087e5a1f758c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.108943 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126f7b0e88774d51ac087e5a1f758c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.108879 4165 tablet_service.cc:1467] Processing CreateTablet for tablet 126f7b0e88774d51ac087e5a1f758c78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.108901 4558 tablet_service.cc:1467] Processing CreateTablet for tablet 7d320136fd1d4eb895362144ef6a5322 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.109284 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126f7b0e88774d51ac087e5a1f758c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.109326 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d320136fd1d4eb895362144ef6a5322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.116025 4429 tablet_service.cc:1467] Processing CreateTablet for tablet 7d320136fd1d4eb895362144ef6a5322 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.116447 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d320136fd1d4eb895362144ef6a5322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.117148 4427 tablet_service.cc:1467] Processing CreateTablet for tablet d48a2b58ce2d417e9abf63c292419d14 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.117522 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48a2b58ce2d417e9abf63c292419d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.118136 5449 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:54.118837 4164 tablet_service.cc:1467] Processing CreateTablet for tablet 7d320136fd1d4eb895362144ef6a5322 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.119282 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d320136fd1d4eb895362144ef6a5322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.119287 4298 tablet_service.cc:1467] Processing CreateTablet for tablet d48a2b58ce2d417e9abf63c292419d14 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.119657 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48a2b58ce2d417e9abf63c292419d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.121469 4560 tablet_service.cc:1467] Processing CreateTablet for tablet d48a2b58ce2d417e9abf63c292419d14 (DEFAULT_TABLE table=rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:54.121899 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48a2b58ce2d417e9abf63c292419d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:54.123217 5449 tablet_bootstrap.cc:654] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.125313 5450 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:54.127204 5450 tablet_bootstrap.cc:654] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.132167 5450 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:54.132388 5450 ts_tablet_manager.cc:1367] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:10:54.133278 5450 raft_consensus.cc:348] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.134428 5452 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:54.134570 5450 raft_consensus.cc:374] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.134889 5450 raft_consensus.cc:729] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.135314 5450 consensus_queue.cc:260] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.136327 5450 ts_tablet_manager.cc:1398] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:54.137866 5452 tablet_bootstrap.cc:654] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.146775 5452 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:54.146847 5450 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:54.146992 5452 ts_tablet_manager.cc:1367] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240501 20:10:54.147908 5452 raft_consensus.cc:348] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.148156 5452 raft_consensus.cc:374] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.148252 5452 raft_consensus.cc:729] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.148524 5452 consensus_queue.cc:260] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.149123 5450 tablet_bootstrap.cc:654] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.149168 5452 ts_tablet_manager.cc:1398] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:54.157699 5452 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:54.159183 5448 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:54.159518 5452 tablet_bootstrap.cc:654] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.160497 5448 tablet_bootstrap.cc:654] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.161556 5449 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:54.161732 5449 ts_tablet_manager.cc:1367] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.015s
I20240501 20:10:54.162460 5452 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:54.162669 5448 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:54.162691 5452 ts_tablet_manager.cc:1367] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 20:10:54.162665 5449 raft_consensus.cc:348] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.162905 5449 raft_consensus.cc:374] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.163007 5449 raft_consensus.cc:729] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.162956 5448 ts_tablet_manager.cc:1367] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:54.163304 5449 consensus_queue.cc:260] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.163545 5450 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:54.163447 5452 raft_consensus.cc:348] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.163662 5452 raft_consensus.cc:374] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.163746 5452 raft_consensus.cc:729] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.164011 5449 ts_tablet_manager.cc:1398] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 20:10:54.163985 5452 consensus_queue.cc:260] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.164260 5448 raft_consensus.cc:348] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.164507 5448 raft_consensus.cc:374] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.164640 5448 raft_consensus.cc:729] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.164829 5452 ts_tablet_manager.cc:1398] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:54.164984 5449 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:54.164391 5450 ts_tablet_manager.cc:1367] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20240501 20:10:54.165012 5448 consensus_queue.cc:260] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.166098 5450 raft_consensus.cc:348] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.166353 5450 raft_consensus.cc:374] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.166428 5450 raft_consensus.cc:729] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.166594 5449 tablet_bootstrap.cc:654] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.166667 5450 consensus_queue.cc:260] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.167493 5450 ts_tablet_manager.cc:1398] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:54.169821 5448 ts_tablet_manager.cc:1398] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20240501 20:10:54.170167 5448 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:54.170761 5449 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:54.170933 5449 ts_tablet_manager.cc:1367] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:10:54.171711 5449 raft_consensus.cc:348] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.171936 5449 raft_consensus.cc:374] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.172014 5449 raft_consensus.cc:729] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.172031 5448 tablet_bootstrap.cc:654] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.172268 5449 consensus_queue.cc:260] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.172817 5449 ts_tablet_manager.cc:1398] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:54.174301 5448 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:54.174494 5448 ts_tablet_manager.cc:1367] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 20:10:54.175271 5448 raft_consensus.cc:348] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.175506 5448 raft_consensus.cc:374] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.175590 5448 raft_consensus.cc:729] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.175889 5448 consensus_queue.cc:260] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.176589 5448 ts_tablet_manager.cc:1398] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:54.176916 5448 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:54.178308 5448 tablet_bootstrap.cc:654] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:54.179934 5448 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:54.180083 5448 ts_tablet_manager.cc:1367] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:54.180678 5448 raft_consensus.cc:348] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.180884 5448 raft_consensus.cc:374] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:54.180948 5448 raft_consensus.cc:729] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:54.181130 5448 consensus_queue.cc:260] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.181560 5448 ts_tablet_manager.cc:1398] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 20:10:54.244031 5426 raft_consensus.cc:482] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:54.244221 5426 raft_consensus.cc:504] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.244994 5426 leader_election.cc:288] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:54.245478 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d320136fd1d4eb895362144ef6a5322" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:54.245482 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d320136fd1d4eb895362144ef6a5322" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:54.245707 4579 raft_consensus.cc:2454] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:54.245713 4449 raft_consensus.cc:2454] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:54.246270 4115 leader_election.cc:302] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:54.246541 5426 raft_consensus.cc:2781] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:54.246647 5426 raft_consensus.cc:482] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:54.246745 5426 raft_consensus.cc:3037] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.248814 5426 raft_consensus.cc:504] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.249351 5426 leader_election.cc:288] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:54.249737 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d320136fd1d4eb895362144ef6a5322" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:54.249960 4579 raft_consensus.cc:3037] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.249912 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d320136fd1d4eb895362144ef6a5322" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:54.250120 4449 raft_consensus.cc:3037] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.252764 4579 raft_consensus.cc:2454] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:54.252756 4449 raft_consensus.cc:2454] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:54.253171 4115 leader_election.cc:302] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:54.253445 5426 raft_consensus.cc:2781] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:54.253592 5426 raft_consensus.cc:686] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:54.253881 5426 consensus_queue.cc:237] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:54.256541 4025 catalog_manager.cc:5484] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:54.266635 5426 raft_consensus.cc:482] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:54.266801 5426 raft_consensus.cc:504] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.267362 5426 leader_election.cc:288] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:54.267820 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:54.267851 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:10:54.268069 4317 raft_consensus.cc:2454] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:54.268095 4449 raft_consensus.cc:2454] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:54.268488 4118 leader_election.cc:302] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:54.268811 5426 raft_consensus.cc:2781] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:54.268899 5426 raft_consensus.cc:482] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:54.269001 5426 raft_consensus.cc:3037] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.270812 5426 raft_consensus.cc:504] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.271379 5426 leader_election.cc:288] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:54.271704 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:54.271800 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:10:54.271872 4449 raft_consensus.cc:3037] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.272001 4317 raft_consensus.cc:3037] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.273759 4449 raft_consensus.cc:2454] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:54.273808 4317 raft_consensus.cc:2454] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:54.274130 4115 leader_election.cc:302] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:54.274379 5426 raft_consensus.cc:2781] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:54.274511 5426 raft_consensus.cc:686] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:54.274722 5426 consensus_queue.cc:237] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.277298 4025 catalog_manager.cc:5484] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:54.336503 5291 raft_consensus.cc:482] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:54.336697 5291 raft_consensus.cc:504] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.337649 5291 leader_election.cc:288] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:54.345340 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48a2b58ce2d417e9abf63c292419d14" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:54.345638 4449 raft_consensus.cc:2454] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:10:54.348289 4513 leader_election.cc:302] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:54.348611 5291 raft_consensus.cc:2781] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:54.348803 5291 raft_consensus.cc:482] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:54.348919 5291 raft_consensus.cc:3037] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.351802 5291 raft_consensus.cc:504] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.352839 5291 leader_election.cc:288] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:54.352988 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48a2b58ce2d417e9abf63c292419d14" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:54.353195 4448 raft_consensus.cc:3037] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.353315 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48a2b58ce2d417e9abf63c292419d14" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:10:54.353523 4318 raft_consensus.cc:3037] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:54.355198 4448 raft_consensus.cc:2454] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:10:54.355716 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48a2b58ce2d417e9abf63c292419d14" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:10:54.356339 4318 raft_consensus.cc:2454] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:10:54.356786 4515 leader_election.cc:302] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:54.357110 5291 raft_consensus.cc:2781] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:54.357565 5291 raft_consensus.cc:686] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Becoming Leader. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Running, Role: LEADER
I20240501 20:10:54.358055 5291 consensus_queue.cc:237] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:54.361809 4025 catalog_manager.cc:5484] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:54.720505 5426 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:54.727906 5364 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:54.742607 5470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:54.742931 5470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:54.750487 5470 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:54.774899 5426 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 20:10:54.781201 5364 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:54.883162 5291 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:54.910039 5291 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 231.752 ms
time per row: 0.115876 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:10:55.993746 4026 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:56.002605 4298 tablet_service.cc:1467] Processing CreateTablet for tablet c11109bc41e34ab58fe3c9f568adf85e (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.003080 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11109bc41e34ab58fe3c9f568adf85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.003949 4296 tablet_service.cc:1467] Processing CreateTablet for tablet 618ea2abb259425babf56375f41eb81c (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.004011 4558 tablet_service.cc:1467] Processing CreateTablet for tablet 13d3d6724e3242478fef21f83dc94452 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.004369 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ea2abb259425babf56375f41eb81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.004448 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d3d6724e3242478fef21f83dc94452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.004343 4163 tablet_service.cc:1467] Processing CreateTablet for tablet 13d3d6724e3242478fef21f83dc94452 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.004794 4163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d3d6724e3242478fef21f83dc94452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.005474 4165 tablet_service.cc:1467] Processing CreateTablet for tablet c11109bc41e34ab58fe3c9f568adf85e (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.005838 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11109bc41e34ab58fe3c9f568adf85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.008193 4428 tablet_service.cc:1467] Processing CreateTablet for tablet c11109bc41e34ab58fe3c9f568adf85e (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.008584 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11109bc41e34ab58fe3c9f568adf85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.009196 4164 tablet_service.cc:1467] Processing CreateTablet for tablet 618ea2abb259425babf56375f41eb81c (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.009537 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ea2abb259425babf56375f41eb81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.009449 4429 tablet_service.cc:1467] Processing CreateTablet for tablet 13d3d6724e3242478fef21f83dc94452 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.009833 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d3d6724e3242478fef21f83dc94452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.013626 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 618ea2abb259425babf56375f41eb81c (DEFAULT_TABLE table=rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.014094 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ea2abb259425babf56375f41eb81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.020431 5538 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:56.020776 5540 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:56.022413 5538 tablet_bootstrap.cc:654] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.022526 5540 tablet_bootstrap.cc:654] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.023041 5539 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:56.025645 5539 tablet_bootstrap.cc:654] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.032778 5538 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:56.032976 5538 ts_tablet_manager.cc:1367] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240501 20:10:56.033898 5538 raft_consensus.cc:348] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.034138 5538 raft_consensus.cc:374] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.034232 5538 raft_consensus.cc:729] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.034492 5538 consensus_queue.cc:260] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.034988 5539 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:56.035200 5539 ts_tablet_manager.cc:1367] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240501 20:10:56.035781 5538 ts_tablet_manager.cc:1398] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:56.036113 5539 raft_consensus.cc:348] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.036409 5539 raft_consensus.cc:374] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.036554 5539 raft_consensus.cc:729] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.036922 5539 consensus_queue.cc:260] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.038244 5538 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:56.042774 5538 tablet_bootstrap.cc:654] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.044836 5539 ts_tablet_manager.cc:1398] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240501 20:10:56.045291 5539 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:56.047216 5539 tablet_bootstrap.cc:654] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.051750 5538 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:56.052047 5538 ts_tablet_manager.cc:1367] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.009s
I20240501 20:10:56.052393 5539 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:56.052608 5539 ts_tablet_manager.cc:1367] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20240501 20:10:56.052995 5538 raft_consensus.cc:348] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.053337 5538 raft_consensus.cc:374] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.053472 5538 raft_consensus.cc:729] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.053416 5539 raft_consensus.cc:348] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.053687 5539 raft_consensus.cc:374] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.053804 5539 raft_consensus.cc:729] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.053826 5538 consensus_queue.cc:260] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.054114 5539 consensus_queue.cc:260] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.054589 5538 ts_tablet_manager.cc:1398] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:56.054935 5538 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:56.056819 5538 tablet_bootstrap.cc:654] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.057080 5539 ts_tablet_manager.cc:1398] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:56.058710 5538 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:56.058910 5538 ts_tablet_manager.cc:1367] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:56.059849 5538 raft_consensus.cc:348] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.060247 5538 raft_consensus.cc:374] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.060400 5538 raft_consensus.cc:729] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.060420 5545 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:56.060715 5538 consensus_queue.cc:260] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.061538 5538 ts_tablet_manager.cc:1398] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:56.062577 5545 tablet_bootstrap.cc:654] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.064651 5540 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:56.064864 5540 ts_tablet_manager.cc:1367] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.046s user 0.018s sys 0.018s
I20240501 20:10:56.064904 5545 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:56.065038 5545 ts_tablet_manager.cc:1367] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:10:56.065698 5540 raft_consensus.cc:348] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.065793 5545 raft_consensus.cc:348] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.065948 5540 raft_consensus.cc:374] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.065984 5545 raft_consensus.cc:374] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.066054 5540 raft_consensus.cc:729] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.066063 5545 raft_consensus.cc:729] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.066293 5545 consensus_queue.cc:260] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.066332 5540 consensus_queue.cc:260] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.066887 5540 ts_tablet_manager.cc:1398] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:56.066887 5545 ts_tablet_manager.cc:1398] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:56.067209 5545 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:56.067209 5540 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:56.067529 5465 raft_consensus.cc:482] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:56.067684 5465 raft_consensus.cc:504] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.068431 5465 leader_election.cc:288] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:56.069100 5545 tablet_bootstrap.cc:654] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.069957 5540 tablet_bootstrap.cc:654] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.070843 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11109bc41e34ab58fe3c9f568adf85e" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:10:56.071095 4186 raft_consensus.cc:2454] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:56.071501 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11109bc41e34ab58fe3c9f568adf85e" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:56.071697 4449 raft_consensus.cc:2454] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:56.072060 4251 leader_election.cc:302] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:56.072342 5465 raft_consensus.cc:2781] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:56.072456 5465 raft_consensus.cc:482] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:56.072541 5465 raft_consensus.cc:3037] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.075420 5465 raft_consensus.cc:504] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.076117 5540 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:56.076126 5465 leader_election.cc:288] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:56.076300 5545 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:56.076287 5540 ts_tablet_manager.cc:1367] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240501 20:10:56.076511 5545 ts_tablet_manager.cc:1367] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20240501 20:10:56.077103 5540 raft_consensus.cc:348] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.077244 5545 raft_consensus.cc:348] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.077358 5540 raft_consensus.cc:374] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.077427 5540 raft_consensus.cc:729] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.077450 5545 raft_consensus.cc:374] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.077514 5545 raft_consensus.cc:729] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.077690 5540 consensus_queue.cc:260] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.077831 5545 consensus_queue.cc:260] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.078083 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11109bc41e34ab58fe3c9f568adf85e" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:56.078331 4449 raft_consensus.cc:3037] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.078351 5545 ts_tablet_manager.cc:1398] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:56.078346 5540 ts_tablet_manager.cc:1398] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:56.079167 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11109bc41e34ab58fe3c9f568adf85e" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:10:56.079380 4186 raft_consensus.cc:3037] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.080602 4449 raft_consensus.cc:2454] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:56.081030 4251 leader_election.cc:302] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters:
I20240501 20:10:56.081452 5465 raft_consensus.cc:2781] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:56.081698 4186 raft_consensus.cc:2454] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:56.081867 5465 raft_consensus.cc:686] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:10:56.082154 5465 consensus_queue.cc:237] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:56.084894 4029 catalog_manager.cc:5484] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:56.112627 5533 raft_consensus.cc:482] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:56.112850 5533 raft_consensus.cc:504] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.113513 5533 leader_election.cc:288] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:56.114120 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ea2abb259425babf56375f41eb81c" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:56.114209 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ea2abb259425babf56375f41eb81c" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:10:56.114401 4579 raft_consensus.cc:2454] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:56.114437 4318 raft_consensus.cc:2454] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:10:56.114878 4118 leader_election.cc:302] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:56.115134 5533 raft_consensus.cc:2781] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:56.115281 5533 raft_consensus.cc:482] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:56.115367 5533 raft_consensus.cc:3037] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.117262 5533 raft_consensus.cc:504] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.117781 5533 leader_election.cc:288] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:56.118232 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ea2abb259425babf56375f41eb81c" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:56.118271 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ea2abb259425babf56375f41eb81c" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:10:56.118417 4579 raft_consensus.cc:3037] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.118468 4318 raft_consensus.cc:3037] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.120270 4579 raft_consensus.cc:2454] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:56.120615 4318 raft_consensus.cc:2454] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:10:56.120707 4118 leader_election.cc:302] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:56.121044 5533 raft_consensus.cc:2781] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:56.121219 5533 raft_consensus.cc:686] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:10:56.121551 5533 consensus_queue.cc:237] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:56.124334 4029 catalog_manager.cc:5484] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:56.271087 5532 raft_consensus.cc:482] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:56.271276 5532 raft_consensus.cc:504] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.271894 5532 leader_election.cc:288] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:10:56.272485 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d3d6724e3242478fef21f83dc94452" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:56.272526 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d3d6724e3242478fef21f83dc94452" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:10:56.272796 4449 raft_consensus.cc:2454] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:10:56.272814 4186 raft_consensus.cc:2454] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:10:56.273205 4513 leader_election.cc:302] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:56.273535 5532 raft_consensus.cc:2781] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:56.273669 5532 raft_consensus.cc:482] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:56.273753 5532 raft_consensus.cc:3037] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.275583 5532 raft_consensus.cc:504] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.276140 5532 leader_election.cc:288] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:10:56.276651 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d3d6724e3242478fef21f83dc94452" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:56.276718 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d3d6724e3242478fef21f83dc94452" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:10:56.276897 4449 raft_consensus.cc:3037] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.276947 4186 raft_consensus.cc:3037] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.279691 4186 raft_consensus.cc:2454] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:10:56.279703 4449 raft_consensus.cc:2454] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:10:56.280143 4513 leader_election.cc:302] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:56.280447 5532 raft_consensus.cc:2781] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:56.280823 5532 raft_consensus.cc:686] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Becoming Leader. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Running, Role: LEADER
I20240501 20:10:56.281147 5532 consensus_queue.cc:237] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:10:56.283893 4029 catalog_manager.cc:5484] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:56.511417 5465 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:56.516800 5465 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:56.629529 5533 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:56.638046 5426 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:56.685999 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:56.686318 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:56.694356 5557 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:56.718595 5532 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:56.723546 5532 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 267.337 ms
time per row: 0.133669 ms
I20240501 20:10:57.154122 4025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:57.163849 4429 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff47913ab3b4771abbd9c912bbf34ad (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.164345 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff47913ab3b4771abbd9c912bbf34ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.164243 4298 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff47913ab3b4771abbd9c912bbf34ad (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.164695 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff47913ab3b4771abbd9c912bbf34ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.165674 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff47913ab3b4771abbd9c912bbf34ad (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.165933 4428 tablet_service.cc:1467] Processing CreateTablet for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.166090 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff47913ab3b4771abbd9c912bbf34ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.166263 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79cb3f8b3fe94497bb54d3d8b7f18754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.169232 4163 tablet_service.cc:1467] Processing CreateTablet for tablet bbd253f3e7824a81b6fdaf2c8880aaa9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.169296 4427 tablet_service.cc:1467] Processing CreateTablet for tablet bbd253f3e7824a81b6fdaf2c8880aaa9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.169646 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd253f3e7824a81b6fdaf2c8880aaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.169661 4163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd253f3e7824a81b6fdaf2c8880aaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.173141 4558 tablet_service.cc:1467] Processing CreateTablet for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.173537 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79cb3f8b3fe94497bb54d3d8b7f18754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.176788 4296 tablet_service.cc:1467] Processing CreateTablet for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.177222 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79cb3f8b3fe94497bb54d3d8b7f18754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.181180 4297 tablet_service.cc:1467] Processing CreateTablet for tablet bbd253f3e7824a81b6fdaf2c8880aaa9 (DEFAULT_TABLE table=rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.181609 4297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd253f3e7824a81b6fdaf2c8880aaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.184700 5631 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:57.187098 5631 tablet_bootstrap.cc:654] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.193994 5629 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:57.195955 5629 tablet_bootstrap.cc:654] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.196504 5631 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:57.196789 5631 ts_tablet_manager.cc:1367] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.003s
I20240501 20:10:57.197698 5631 raft_consensus.cc:348] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.198032 5631 raft_consensus.cc:374] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.198168 5631 raft_consensus.cc:729] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.198504 5631 consensus_queue.cc:260] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.204095 5629 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:57.204298 5629 ts_tablet_manager.cc:1367] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240501 20:10:57.204335 5631 ts_tablet_manager.cc:1398] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:57.205209 5629 raft_consensus.cc:348] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.205438 5632 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:57.205539 5629 raft_consensus.cc:374] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.205715 5629 raft_consensus.cc:729] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.206079 5629 consensus_queue.cc:260] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.206748 5629 ts_tablet_manager.cc:1398] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:57.207041 5630 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:57.207124 5629 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:57.208972 5629 tablet_bootstrap.cc:654] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.209089 5630 tablet_bootstrap.cc:654] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.210237 5632 tablet_bootstrap.cc:654] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.214346 5631 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:57.215201 5632 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:57.215412 5632 ts_tablet_manager.cc:1367] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240501 20:10:57.216253 5631 tablet_bootstrap.cc:654] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.216641 5632 raft_consensus.cc:348] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.216966 5632 raft_consensus.cc:374] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.217096 5632 raft_consensus.cc:729] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.217412 5632 consensus_queue.cc:260] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.218187 5632 ts_tablet_manager.cc:1398] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:57.218530 5632 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:57.220083 5632 tablet_bootstrap.cc:654] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.221659 5630 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:57.221899 5630 ts_tablet_manager.cc:1367] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 20:10:57.222757 5630 raft_consensus.cc:348] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.223068 5630 raft_consensus.cc:374] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.223201 5630 raft_consensus.cc:729] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.223547 5630 consensus_queue.cc:260] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.224330 5630 ts_tablet_manager.cc:1398] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:57.227399 5629 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:57.227532 5631 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:57.227654 5629 ts_tablet_manager.cc:1367] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.015s
I20240501 20:10:57.227743 5631 ts_tablet_manager.cc:1367] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240501 20:10:57.228533 5631 raft_consensus.cc:348] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.228530 5629 raft_consensus.cc:348] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.228793 5629 raft_consensus.cc:374] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.228875 5631 raft_consensus.cc:374] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.228914 5629 raft_consensus.cc:729] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.228960 5631 raft_consensus.cc:729] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.229300 5629 consensus_queue.cc:260] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.229326 5631 consensus_queue.cc:260] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.230028 5629 ts_tablet_manager.cc:1398] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:57.233487 5632 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:57.233754 5632 ts_tablet_manager.cc:1367] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.009s
I20240501 20:10:57.234645 5632 raft_consensus.cc:348] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.234851 5631 ts_tablet_manager.cc:1398] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 20:10:57.234902 5632 raft_consensus.cc:374] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.234969 5632 raft_consensus.cc:729] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.235216 5632 consensus_queue.cc:260] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.235767 5632 ts_tablet_manager.cc:1398] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:57.236172 5632 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:57.237912 5632 tablet_bootstrap.cc:654] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.238839 5631 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:57.240846 5631 tablet_bootstrap.cc:654] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.242532 5632 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:57.242678 5632 ts_tablet_manager.cc:1367] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240501 20:10:57.242910 5631 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:57.243063 5631 ts_tablet_manager.cc:1367] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 20:10:57.243373 5632 raft_consensus.cc:348] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.243542 5632 raft_consensus.cc:374] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.243669 5632 raft_consensus.cc:729] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.243824 5631 raft_consensus.cc:348] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.243891 5632 consensus_queue.cc:260] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.244072 5631 raft_consensus.cc:374] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.244184 5631 raft_consensus.cc:729] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.244464 5631 consensus_queue.cc:260] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.245066 5631 ts_tablet_manager.cc:1398] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:57.246054 5632 ts_tablet_manager.cc:1398] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:57.258169 5615 raft_consensus.cc:482] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:57.258342 5615 raft_consensus.cc:504] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.259121 5615 leader_election.cc:288] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:57.259559 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cb3f8b3fe94497bb54d3d8b7f18754" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:57.259681 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cb3f8b3fe94497bb54d3d8b7f18754" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:57.259809 4579 raft_consensus.cc:2454] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:57.259941 4448 raft_consensus.cc:2454] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:57.260262 4253 leader_election.cc:302] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:57.260540 5615 raft_consensus.cc:2781] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:57.260654 5615 raft_consensus.cc:482] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:57.260778 5615 raft_consensus.cc:3037] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.262764 5615 raft_consensus.cc:504] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.263370 5615 leader_election.cc:288] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:57.263788 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cb3f8b3fe94497bb54d3d8b7f18754" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:57.263825 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79cb3f8b3fe94497bb54d3d8b7f18754" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:57.263989 4579 raft_consensus.cc:3037] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.264024 4448 raft_consensus.cc:3037] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.264667 5615 raft_consensus.cc:482] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:57.264823 5615 raft_consensus.cc:504] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.265811 5615 leader_election.cc:288] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:57.265836 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:57.266050 4449 raft_consensus.cc:2454] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:57.266350 4448 raft_consensus.cc:2454] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:57.266419 4579 raft_consensus.cc:2454] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:57.266496 4251 leader_election.cc:302] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters:
I20240501 20:10:57.266722 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:57.266804 4251 leader_election.cc:302] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters:
I20240501 20:10:57.266925 4580 raft_consensus.cc:2454] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:57.266919 5615 raft_consensus.cc:2781] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:57.267050 5615 raft_consensus.cc:482] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:57.267130 5615 raft_consensus.cc:3037] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.267220 5465 raft_consensus.cc:2781] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:57.267396 5465 raft_consensus.cc:686] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:10:57.267660 5465 consensus_queue.cc:237] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:57.269698 5615 raft_consensus.cc:504] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.270198 5615 leader_election.cc:288] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:57.270570 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:57.270725 4448 raft_consensus.cc:3037] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.270669 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:57.270855 4580 raft_consensus.cc:3037] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.271252 4025 catalog_manager.cc:5484] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:57.272634 4448 raft_consensus.cc:2454] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:57.273008 4251 leader_election.cc:302] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters:
I20240501 20:10:57.273456 4580 raft_consensus.cc:2454] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:57.274295 5615 raft_consensus.cc:2781] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:57.274642 5615 raft_consensus.cc:686] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:10:57.274909 5615 consensus_queue.cc:237] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:57.277639 4025 catalog_manager.cc:5484] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:57.430994 5615 raft_consensus.cc:482] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:57.431646 5615 raft_consensus.cc:504] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.432358 5615 leader_election.cc:288] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:57.432912 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:10:57.433161 4186 raft_consensus.cc:2454] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:57.433152 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:10:57.433598 4448 raft_consensus.cc:2454] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:57.433612 4251 leader_election.cc:302] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:57.433929 5615 raft_consensus.cc:2781] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:57.434033 5615 raft_consensus.cc:482] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:57.434139 5615 raft_consensus.cc:3037] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.436255 5615 raft_consensus.cc:504] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.436892 5615 leader_election.cc:288] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:10:57.437449 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:10:57.437659 4186 raft_consensus.cc:3037] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.437723 4449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:10:57.437963 4449 raft_consensus.cc:3037] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.439848 4186 raft_consensus.cc:2454] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:57.440282 4251 leader_election.cc:302] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:57.440528 5615 raft_consensus.cc:2781] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:57.440671 5615 raft_consensus.cc:686] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:10:57.440755 4449 raft_consensus.cc:2454] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:57.440975 5615 consensus_queue.cc:237] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:10:57.444207 4025 catalog_manager.cc:5484] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:57.664716 5465 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:57.669855 5615 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:57.724009 5615 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:57.731087 5552 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:57.848984 5646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:57.849341 5646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:57.856971 5646 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 20:10:57.998965 4186 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:57.999768 4447 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:57.999784 5615 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:58.000772 5615 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 199.673 ms
time per row: 0.0998366 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:10:59.067332 4025 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:59.076434 4163 tablet_service.cc:1467] Processing CreateTablet for tablet 0822a3fc82134d018bae96fecc62fdd3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.076996 4163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0822a3fc82134d018bae96fecc62fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.077278 4429 tablet_service.cc:1467] Processing CreateTablet for tablet bb36776ab8b74586aead4ecad56cc815 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.077656 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb36776ab8b74586aead4ecad56cc815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.080008 4164 tablet_service.cc:1467] Processing CreateTablet for tablet bbf736b8dba9474da5fb7ca9d9348ac7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.080420 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf736b8dba9474da5fb7ca9d9348ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.081991 5712 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:59.082794 5713 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:10:59.083676 5712 tablet_bootstrap.cc:654] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.084285 5713 tablet_bootstrap.cc:654] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.085131 4165 tablet_service.cc:1467] Processing CreateTablet for tablet bb36776ab8b74586aead4ecad56cc815 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.085495 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb36776ab8b74586aead4ecad56cc815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.085739 5712 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:59.085877 5712 ts_tablet_manager.cc:1367] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:59.085881 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 0822a3fc82134d018bae96fecc62fdd3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.086282 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0822a3fc82134d018bae96fecc62fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.086712 5712 raft_consensus.cc:348] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.086948 5712 raft_consensus.cc:374] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.087040 5712 raft_consensus.cc:729] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.087313 5712 consensus_queue.cc:260] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.087841 5712 ts_tablet_manager.cc:1398] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:59.089984 4298 tablet_service.cc:1467] Processing CreateTablet for tablet bbf736b8dba9474da5fb7ca9d9348ac7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.089985 4559 tablet_service.cc:1467] Processing CreateTablet for tablet bbf736b8dba9474da5fb7ca9d9348ac7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.090390 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf736b8dba9474da5fb7ca9d9348ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.090411 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf736b8dba9474da5fb7ca9d9348ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.090368 4558 tablet_service.cc:1467] Processing CreateTablet for tablet bb36776ab8b74586aead4ecad56cc815 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.090713 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb36776ab8b74586aead4ecad56cc815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.089985 4296 tablet_service.cc:1467] Processing CreateTablet for tablet 0822a3fc82134d018bae96fecc62fdd3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:59.094656 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0822a3fc82134d018bae96fecc62fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:59.097525 5716 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:59.097880 5713 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:10:59.099161 5716 tablet_bootstrap.cc:654] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.102166 5713 ts_tablet_manager.cc:1367] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20240501 20:10:59.102953 5712 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:59.103456 5713 raft_consensus.cc:348] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.103804 5713 raft_consensus.cc:374] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.103941 5713 raft_consensus.cc:729] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.104264 5713 consensus_queue.cc:260] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.104590 5712 tablet_bootstrap.cc:654] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.109144 5717 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:59.109727 5712 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:59.109895 5712 ts_tablet_manager.cc:1367] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:10:59.110520 5717 tablet_bootstrap.cc:654] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.110947 5712 raft_consensus.cc:348] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.111186 5712 raft_consensus.cc:374] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.111331 5712 raft_consensus.cc:729] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.111652 5712 consensus_queue.cc:260] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.111878 5713 ts_tablet_manager.cc:1398] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20240501 20:10:59.112270 5712 ts_tablet_manager.cc:1398] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:59.112843 5717 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:59.112839 5716 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:59.112977 5717 ts_tablet_manager.cc:1367] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:59.112993 5716 ts_tablet_manager.cc:1367] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20240501 20:10:59.113180 5712 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:10:59.113739 5716 raft_consensus.cc:348] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.113732 5717 raft_consensus.cc:348] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.113935 5717 raft_consensus.cc:374] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.113935 5716 raft_consensus.cc:374] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.114025 5716 raft_consensus.cc:729] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.114137 5717 raft_consensus.cc:729] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.114243 5716 consensus_queue.cc:260] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.114485 5717 consensus_queue.cc:260] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.114742 5716 ts_tablet_manager.cc:1398] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:59.115020 5716 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:59.115741 5717 ts_tablet_manager.cc:1398] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:59.116000 5717 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:10:59.117076 5716 tablet_bootstrap.cc:654] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.114830 5712 tablet_bootstrap.cc:654] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.120653 5717 tablet_bootstrap.cc:654] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.124390 5712 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:10:59.124459 5717 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:10:59.124550 5712 ts_tablet_manager.cc:1367] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 20:10:59.124605 5717 ts_tablet_manager.cc:1367] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:10:59.124725 5716 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:59.124953 5716 ts_tablet_manager.cc:1367] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240501 20:10:59.125365 5717 raft_consensus.cc:348] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.125566 5717 raft_consensus.cc:374] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.125661 5717 raft_consensus.cc:729] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.125643 5716 raft_consensus.cc:348] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.125662 5712 raft_consensus.cc:348] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.125921 5712 raft_consensus.cc:374] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.125918 5716 raft_consensus.cc:374] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.125995 5716 raft_consensus.cc:729] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.125936 5717 consensus_queue.cc:260] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.126044 5712 raft_consensus.cc:729] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.126242 5716 consensus_queue.cc:260] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.126287 5712 consensus_queue.cc:260] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.126580 5717 ts_tablet_manager.cc:1398] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:59.126966 5716 ts_tablet_manager.cc:1398] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:10:59.127269 5716 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:10:59.129050 5716 tablet_bootstrap.cc:654] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:59.129204 5712 ts_tablet_manager.cc:1398] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:59.132611 5716 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:10:59.132911 5716 ts_tablet_manager.cc:1367] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240501 20:10:59.133697 5716 raft_consensus.cc:348] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.133947 5716 raft_consensus.cc:374] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:59.134055 5716 raft_consensus.cc:729] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:10:59.134272 5716 consensus_queue.cc:260] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.134886 5716 ts_tablet_manager.cc:1398] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:59.173662 5555 raft_consensus.cc:482] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:59.173880 5555 raft_consensus.cc:504] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.174628 5555 leader_election.cc:288] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:59.175118 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf736b8dba9474da5fb7ca9d9348ac7" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:10:59.175135 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf736b8dba9474da5fb7ca9d9348ac7" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:10:59.175335 4318 raft_consensus.cc:2454] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:10:59.175369 4186 raft_consensus.cc:2454] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:10:59.175797 4515 leader_election.cc:302] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:59.176098 5555 raft_consensus.cc:2781] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:59.176218 5555 raft_consensus.cc:482] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:59.176319 5555 raft_consensus.cc:3037] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.178210 5555 raft_consensus.cc:504] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.178762 5555 leader_election.cc:288] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:10:59.179193 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf736b8dba9474da5fb7ca9d9348ac7" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:10:59.179261 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbf736b8dba9474da5fb7ca9d9348ac7" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:10:59.179399 4186 raft_consensus.cc:3037] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.179422 4318 raft_consensus.cc:3037] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.181335 4318 raft_consensus.cc:2454] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:10:59.181416 4186 raft_consensus.cc:2454] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:10:59.181788 4513 leader_election.cc:302] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:59.182101 5555 raft_consensus.cc:2781] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:59.182444 5555 raft_consensus.cc:686] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Becoming Leader. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Running, Role: LEADER
I20240501 20:10:59.182780 5555 consensus_queue.cc:237] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.185709 4031 catalog_manager.cc:5484] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:59.222177 5720 raft_consensus.cc:482] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:59.222378 5720 raft_consensus.cc:504] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.223881 5720 leader_election.cc:288] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:59.243216 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb36776ab8b74586aead4ecad56cc815" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:10:59.243481 4186 raft_consensus.cc:2454] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 0.
I20240501 20:10:59.243588 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb36776ab8b74586aead4ecad56cc815" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:59.243841 4580 raft_consensus.cc:2454] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 0.
I20240501 20:10:59.243959 4382 leader_election.cc:302] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:59.244361 5720 raft_consensus.cc:2781] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:59.244473 5720 raft_consensus.cc:482] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:59.244570 5720 raft_consensus.cc:3037] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.246644 5720 raft_consensus.cc:504] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.247344 5720 leader_election.cc:288] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:59.247738 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb36776ab8b74586aead4ecad56cc815" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:10:59.247946 4186 raft_consensus.cc:3037] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.247888 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb36776ab8b74586aead4ecad56cc815" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:59.248121 4580 raft_consensus.cc:3037] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.249908 4186 raft_consensus.cc:2454] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 1.
I20240501 20:10:59.250043 4580 raft_consensus.cc:2454] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 1.
I20240501 20:10:59.250303 4382 leader_election.cc:302] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:59.250622 5720 raft_consensus.cc:2781] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:59.250972 5720 raft_consensus.cc:686] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Becoming Leader. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Running, Role: LEADER
I20240501 20:10:59.251397 5720 consensus_queue.cc:237] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:10:59.255038 4031 catalog_manager.cc:5484] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 596d3e24651b4787916397f573c09f1d (127.28.136.67). New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:59.553007 5654 raft_consensus.cc:482] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:59.553179 5654 raft_consensus.cc:504] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.553786 5654 leader_election.cc:288] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:59.554260 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0822a3fc82134d018bae96fecc62fdd3" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:10:59.554302 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0822a3fc82134d018bae96fecc62fdd3" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:10:59.554476 4186 raft_consensus.cc:2454] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:59.554494 4580 raft_consensus.cc:2454] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:10:59.554872 4251 leader_election.cc:302] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:10:59.555150 5654 raft_consensus.cc:2781] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:59.555266 5654 raft_consensus.cc:482] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:59.555357 5654 raft_consensus.cc:3037] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.557382 5654 raft_consensus.cc:504] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.557938 5654 leader_election.cc:288] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:10:59.558373 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0822a3fc82134d018bae96fecc62fdd3" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:10:59.558435 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0822a3fc82134d018bae96fecc62fdd3" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:10:59.558574 4186 raft_consensus.cc:3037] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.558621 4580 raft_consensus.cc:3037] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:59.561379 4186 raft_consensus.cc:2454] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:59.561379 4580 raft_consensus.cc:2454] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:10:59.561800 4253 leader_election.cc:302] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:10:59.562105 5654 raft_consensus.cc:2781] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:59.562511 5654 raft_consensus.cc:686] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:10:59.562911 5654 consensus_queue.cc:237] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:10:59.565769 4031 catalog_manager.cc:5484] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:59.715086 5555 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:59.719880 5555 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:59.786598 5744 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:59.794154 5744 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:00.020262 5736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:00.020555 5736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:00.027874 5736 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 20:11:00.150154 5654 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:00.153187 4580 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:00.153868 5654 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 224.679 ms
time per row: 0.11234 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:11:01.258782 4028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:01.268168 4429 tablet_service.cc:1467] Processing CreateTablet for tablet 60871b6059c249bfaac713e2f7325811 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.268550 4164 tablet_service.cc:1467] Processing CreateTablet for tablet 60871b6059c249bfaac713e2f7325811 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.268703 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60871b6059c249bfaac713e2f7325811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.269017 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60871b6059c249bfaac713e2f7325811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.269407 4165 tablet_service.cc:1467] Processing CreateTablet for tablet ae0fe7a1daab476391d80406766b3b11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.269722 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0fe7a1daab476391d80406766b3b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.271018 4427 tablet_service.cc:1467] Processing CreateTablet for tablet 28b9b57213a64172afdc2ca890569fc5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.271415 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b9b57213a64172afdc2ca890569fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.275882 4558 tablet_service.cc:1467] Processing CreateTablet for tablet ae0fe7a1daab476391d80406766b3b11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.276343 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0fe7a1daab476391d80406766b3b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.277287 4298 tablet_service.cc:1467] Processing CreateTablet for tablet 60871b6059c249bfaac713e2f7325811 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.277755 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60871b6059c249bfaac713e2f7325811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.280401 4296 tablet_service.cc:1467] Processing CreateTablet for tablet 28b9b57213a64172afdc2ca890569fc5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.280889 4560 tablet_service.cc:1467] Processing CreateTablet for tablet 28b9b57213a64172afdc2ca890569fc5 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.281272 5813 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:11:01.281299 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b9b57213a64172afdc2ca890569fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.281442 4428 tablet_service.cc:1467] Processing CreateTablet for tablet ae0fe7a1daab476391d80406766b3b11 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:01.281886 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0fe7a1daab476391d80406766b3b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.284354 5813 tablet_bootstrap.cc:654] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.286124 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b9b57213a64172afdc2ca890569fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:01.296293 5816 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:11:01.300194 5813 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:11:01.300360 5813 ts_tablet_manager.cc:1367] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240501 20:11:01.300899 5816 tablet_bootstrap.cc:654] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.300951 5817 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:11:01.301326 5814 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:11:01.302632 5817 tablet_bootstrap.cc:654] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.303059 5814 tablet_bootstrap.cc:654] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.303596 5813 raft_consensus.cc:348] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.303826 5813 raft_consensus.cc:374] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.303923 5813 raft_consensus.cc:729] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.305454 5813 consensus_queue.cc:260] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.306849 5813 ts_tablet_manager.cc:1398] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:11:01.307336 5813 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:11:01.307876 5817 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:11:01.308058 5817 ts_tablet_manager.cc:1367] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:11:01.308882 5817 raft_consensus.cc:348] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.309108 5817 raft_consensus.cc:374] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.309198 5817 raft_consensus.cc:729] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.309330 5813 tablet_bootstrap.cc:654] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.309439 5817 consensus_queue.cc:260] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.310015 5817 ts_tablet_manager.cc:1398] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:01.310895 5816 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:11:01.311081 5816 ts_tablet_manager.cc:1367] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240501 20:11:01.311906 5816 raft_consensus.cc:348] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.312144 5816 raft_consensus.cc:374] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.312238 5816 raft_consensus.cc:729] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.312489 5816 consensus_queue.cc:260] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.313059 5816 ts_tablet_manager.cc:1398] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:11:01.313498 5816 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:11:01.313700 5814 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:11:01.313870 5814 ts_tablet_manager.cc:1367] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20240501 20:11:01.314622 5814 raft_consensus.cc:348] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.314843 5814 raft_consensus.cc:374] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.314941 5814 raft_consensus.cc:729] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.315171 5814 consensus_queue.cc:260] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.315510 5816 tablet_bootstrap.cc:654] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.315768 5814 ts_tablet_manager.cc:1398] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:01.316061 5814 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:11:01.317762 5814 tablet_bootstrap.cc:654] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.320286 5813 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:11:01.320528 5813 ts_tablet_manager.cc:1367] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20240501 20:11:01.321708 5813 raft_consensus.cc:348] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.322027 5813 raft_consensus.cc:374] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.322161 5813 raft_consensus.cc:729] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.322942 5813 consensus_queue.cc:260] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.323544 5813 ts_tablet_manager.cc:1398] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:01.323935 5813 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:11:01.325568 5813 tablet_bootstrap.cc:654] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.327896 5814 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:11:01.327980 5816 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:11:01.328119 5814 ts_tablet_manager.cc:1367] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.010s
I20240501 20:11:01.328166 5816 ts_tablet_manager.cc:1367] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20240501 20:11:01.328950 5816 raft_consensus.cc:348] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.328949 5814 raft_consensus.cc:348] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.329214 5816 raft_consensus.cc:374] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.329336 5816 raft_consensus.cc:729] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.329641 5814 raft_consensus.cc:374] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.329603 5816 consensus_queue.cc:260] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.329757 5814 raft_consensus.cc:729] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.329984 5814 consensus_queue.cc:260] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.330181 5816 ts_tablet_manager.cc:1398] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:01.330626 5814 ts_tablet_manager.cc:1398] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:01.333554 5813 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:11:01.333787 5813 ts_tablet_manager.cc:1367] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20240501 20:11:01.334565 5813 raft_consensus.cc:348] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.330405 5817 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:11:01.334825 5813 raft_consensus.cc:374] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.334934 5813 raft_consensus.cc:729] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.335472 5813 consensus_queue.cc:260] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.336095 5813 ts_tablet_manager.cc:1398] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:11:01.336318 5817 tablet_bootstrap.cc:654] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:01.341066 5817 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:11:01.341245 5817 ts_tablet_manager.cc:1367] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 20:11:01.341991 5817 raft_consensus.cc:348] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.342176 5817 raft_consensus.cc:374] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:01.342248 5817 raft_consensus.cc:729] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:11:01.342474 5817 consensus_queue.cc:260] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.342904 5817 ts_tablet_manager.cc:1398] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:01.459211 5790 raft_consensus.cc:482] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:01.459424 5790 raft_consensus.cc:504] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.460162 5790 leader_election.cc:288] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:01.460767 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b9b57213a64172afdc2ca890569fc5" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:11:01.460788 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b9b57213a64172afdc2ca890569fc5" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:11:01.461024 4579 raft_consensus.cc:2454] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:11:01.461052 4447 raft_consensus.cc:2454] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:11:01.461480 4253 leader_election.cc:302] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:01.461771 5790 raft_consensus.cc:2781] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:01.461880 5790 raft_consensus.cc:482] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:01.461949 5790 raft_consensus.cc:3037] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.464184 5790 raft_consensus.cc:504] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.464843 5790 leader_election.cc:288] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:01.465277 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b9b57213a64172afdc2ca890569fc5" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:11:01.465477 4579 raft_consensus.cc:3037] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.465457 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b9b57213a64172afdc2ca890569fc5" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:11:01.465684 4447 raft_consensus.cc:3037] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.467814 4579 raft_consensus.cc:2454] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:11:01.468011 4447 raft_consensus.cc:2454] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:11:01.468211 4253 leader_election.cc:302] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:01.468528 5790 raft_consensus.cc:2781] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:01.468664 5790 raft_consensus.cc:686] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:11:01.468931 5790 consensus_queue.cc:237] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.471616 4031 catalog_manager.cc:5484] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:01.537845 5555 raft_consensus.cc:482] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:01.538103 5555 raft_consensus.cc:504] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.538797 5555 leader_election.cc:288] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:01.539402 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0fe7a1daab476391d80406766b3b11" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:11:01.539462 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0fe7a1daab476391d80406766b3b11" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:11:01.539623 4447 raft_consensus.cc:2454] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:11:01.539731 4184 raft_consensus.cc:2454] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:11:01.540244 4513 leader_election.cc:302] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:01.540552 5555 raft_consensus.cc:2781] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:01.540660 5555 raft_consensus.cc:482] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:01.540822 5555 raft_consensus.cc:3037] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.542687 5555 raft_consensus.cc:504] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.543284 5555 leader_election.cc:288] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:01.543735 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0fe7a1daab476391d80406766b3b11" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:11:01.543823 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae0fe7a1daab476391d80406766b3b11" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:01.543951 4447 raft_consensus.cc:3037] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.544018 4184 raft_consensus.cc:3037] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.546766 4447 raft_consensus.cc:2454] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:11:01.546775 4184 raft_consensus.cc:2454] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:11:01.547192 4513 leader_election.cc:302] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:01.547464 5555 raft_consensus.cc:2781] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:01.547824 5555 raft_consensus.cc:686] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Becoming Leader. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Running, Role: LEADER
I20240501 20:11:01.548087 5555 consensus_queue.cc:237] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } }
I20240501 20:11:01.548310 5815 raft_consensus.cc:482] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:01.548455 5815 raft_consensus.cc:504] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.549197 5815 leader_election.cc:288] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:01.549801 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60871b6059c249bfaac713e2f7325811" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:11:01.550004 4318 raft_consensus.cc:2454] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:11:01.550405 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60871b6059c249bfaac713e2f7325811" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:11:01.550436 4118 leader_election.cc:302] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:11:01.550612 4447 raft_consensus.cc:2454] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:11:01.550729 5815 raft_consensus.cc:2781] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:01.550830 5815 raft_consensus.cc:482] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:01.550909 5815 raft_consensus.cc:3037] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.551649 4031 catalog_manager.cc:5484] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:01.553262 5815 raft_consensus.cc:504] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.554209 5815 leader_election.cc:288] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:01.554620 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60871b6059c249bfaac713e2f7325811" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:11:01.554800 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60871b6059c249bfaac713e2f7325811" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:01.554901 4447 raft_consensus.cc:3037] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.555033 4318 raft_consensus.cc:3037] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:01.556757 4447 raft_consensus.cc:2454] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:11:01.556864 4318 raft_consensus.cc:2454] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:11:01.557139 4115 leader_election.cc:302] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:11:01.557458 5815 raft_consensus.cc:2781] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:01.557585 5815 raft_consensus.cc:686] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:11:01.557821 5815 consensus_queue.cc:237] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:01.560441 4031 catalog_manager.cc:5484] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:01.927354 5831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:01.927666 5831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:01.935544 5831 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:01.950187 5828 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:01.957269 5555 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240501 20:11:02.018302 5790 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:02.031208 5808 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:02.057512 5815 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:02.187255 5882 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 273.759 ms
time per row: 0.136879 ms
I20240501 20:11:02.363301 4027 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:02.373812 4165 tablet_service.cc:1467] Processing CreateTablet for tablet 96005b2bfa9847bb992640ad40a68b36 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.374053 4296 tablet_service.cc:1467] Processing CreateTablet for tablet 96005b2bfa9847bb992640ad40a68b36 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.374294 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96005b2bfa9847bb992640ad40a68b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.374495 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96005b2bfa9847bb992640ad40a68b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.374706 4298 tablet_service.cc:1467] Processing CreateTablet for tablet d1e01586b0ba42869481577ceacc84e8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.375057 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e01586b0ba42869481577ceacc84e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.379907 5901 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:11:02.380147 4429 tablet_service.cc:1467] Processing CreateTablet for tablet 96005b2bfa9847bb992640ad40a68b36 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.380606 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96005b2bfa9847bb992640ad40a68b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.381794 5901 tablet_bootstrap.cc:654] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.385222 4560 tablet_service.cc:1467] Processing CreateTablet for tablet d1e01586b0ba42869481577ceacc84e8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.385682 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e01586b0ba42869481577ceacc84e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.390385 4165 tablet_service.cc:1467] Processing CreateTablet for tablet d1e01586b0ba42869481577ceacc84e8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.390832 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e01586b0ba42869481577ceacc84e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.394418 4558 tablet_service.cc:1467] Processing CreateTablet for tablet c3b4a011c5944a1db9dbc496e87d5768 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.394870 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b4a011c5944a1db9dbc496e87d5768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.401813 5901 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:11:02.402022 5901 ts_tablet_manager.cc:1367] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.003s
I20240501 20:11:02.402873 5901 raft_consensus.cc:348] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.403244 4164 tablet_service.cc:1467] Processing CreateTablet for tablet c3b4a011c5944a1db9dbc496e87d5768 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.403833 5901 raft_consensus.cc:374] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.403935 5901 raft_consensus.cc:729] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.404104 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b4a011c5944a1db9dbc496e87d5768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.404167 5901 consensus_queue.cc:260] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.405776 5901 ts_tablet_manager.cc:1398] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:02.406273 5904 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:11:02.407794 5904 tablet_bootstrap.cc:654] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.409644 5904 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:11:02.409791 5904 ts_tablet_manager.cc:1367] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:02.410717 5901 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:11:02.410907 5904 raft_consensus.cc:348] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.411104 5904 raft_consensus.cc:374] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.411182 5904 raft_consensus.cc:729] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.411445 5904 consensus_queue.cc:260] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.412338 5901 tablet_bootstrap.cc:654] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.413092 5904 ts_tablet_manager.cc:1398] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:02.413583 4428 tablet_service.cc:1467] Processing CreateTablet for tablet c3b4a011c5944a1db9dbc496e87d5768 (DEFAULT_TABLE table=rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.414049 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b4a011c5944a1db9dbc496e87d5768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.415462 5906 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:11:02.416587 5904 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:11:02.418102 5902 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:11:02.418538 5904 tablet_bootstrap.cc:654] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.419827 5902 tablet_bootstrap.cc:654] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.422178 5906 tablet_bootstrap.cc:654] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.424306 5904 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:11:02.424490 5904 ts_tablet_manager.cc:1367] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:11:02.425314 5904 raft_consensus.cc:348] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.425767 5904 raft_consensus.cc:374] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.425889 5904 raft_consensus.cc:729] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.426187 5904 consensus_queue.cc:260] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.426955 5904 ts_tablet_manager.cc:1398] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:11:02.432519 5901 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:11:02.432821 5901 ts_tablet_manager.cc:1367] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.003s
I20240501 20:11:02.433774 5901 raft_consensus.cc:348] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.434063 5901 raft_consensus.cc:374] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.434167 5901 raft_consensus.cc:729] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.434490 5901 consensus_queue.cc:260] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.437388 5901 ts_tablet_manager.cc:1398] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:11:02.437755 5901 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:11:02.441077 5901 tablet_bootstrap.cc:654] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.442736 5906 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:11:02.442971 5906 ts_tablet_manager.cc:1367] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.009s
I20240501 20:11:02.443809 5902 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:11:02.444020 5902 ts_tablet_manager.cc:1367] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.009s
I20240501 20:11:02.443907 5906 raft_consensus.cc:348] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.444114 5906 raft_consensus.cc:374] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.444159 5901 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:11:02.444299 5901 ts_tablet_manager.cc:1367] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:11:02.444295 5906 raft_consensus.cc:729] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.444576 5906 consensus_queue.cc:260] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.444883 5902 raft_consensus.cc:348] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.445137 5902 raft_consensus.cc:374] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.445256 5902 raft_consensus.cc:729] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.445175 5901 raft_consensus.cc:348] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.445375 5901 raft_consensus.cc:374] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.445526 5901 raft_consensus.cc:729] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.445585 5902 consensus_queue.cc:260] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.445833 5901 consensus_queue.cc:260] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.446564 5901 ts_tablet_manager.cc:1398] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:02.447098 5902 ts_tablet_manager.cc:1398] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:02.447436 5902 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:11:02.448303 5906 ts_tablet_manager.cc:1398] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:11:02.448612 5906 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:11:02.450047 5902 tablet_bootstrap.cc:654] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.450258 5906 tablet_bootstrap.cc:654] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.458878 5902 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:11:02.458950 5906 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:11:02.459080 5902 ts_tablet_manager.cc:1367] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20240501 20:11:02.459095 5906 ts_tablet_manager.cc:1367] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240501 20:11:02.459867 5906 raft_consensus.cc:348] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.459869 5902 raft_consensus.cc:348] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.460111 5902 raft_consensus.cc:374] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.460111 5906 raft_consensus.cc:374] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.460193 5906 raft_consensus.cc:729] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.460203 5902 raft_consensus.cc:729] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.460444 5902 consensus_queue.cc:260] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.460444 5906 consensus_queue.cc:260] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.460997 5902 ts_tablet_manager.cc:1398] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:11:02.461633 5906 ts_tablet_manager.cc:1398] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:11:02.540097 5815 raft_consensus.cc:482] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:02.540331 5815 raft_consensus.cc:504] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.541015 5815 leader_election.cc:288] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:02.541572 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96005b2bfa9847bb992640ad40a68b36" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:11:02.541603 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96005b2bfa9847bb992640ad40a68b36" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:11:02.541805 4318 raft_consensus.cc:2454] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:11:02.541836 4447 raft_consensus.cc:2454] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 0.
I20240501 20:11:02.542241 4118 leader_election.cc:302] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:11:02.542541 5815 raft_consensus.cc:2781] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:02.542659 5815 raft_consensus.cc:482] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:02.542775 5815 raft_consensus.cc:3037] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.544654 5815 raft_consensus.cc:504] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.545213 5815 leader_election.cc:288] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:02.545753 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96005b2bfa9847bb992640ad40a68b36" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:02.545744 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96005b2bfa9847bb992640ad40a68b36" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:11:02.545970 4318 raft_consensus.cc:3037] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.545970 4447 raft_consensus.cc:3037] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.548713 4318 raft_consensus.cc:2454] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:11:02.548713 4447 raft_consensus.cc:2454] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 1.
I20240501 20:11:02.549229 4115 leader_election.cc:302] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:11:02.549266 5555 raft_consensus.cc:482] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:02.549434 5555 raft_consensus.cc:504] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.549547 5815 raft_consensus.cc:2781] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:02.549692 5815 raft_consensus.cc:686] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:11:02.550019 5815 consensus_queue.cc:237] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.550222 5555 leader_election.cc:288] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:02.550637 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e01586b0ba42869481577ceacc84e8" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:11:02.550683 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e01586b0ba42869481577ceacc84e8" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:11:02.550879 4318 raft_consensus.cc:2454] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:11:02.550884 4184 raft_consensus.cc:2454] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 0.
I20240501 20:11:02.551339 4515 leader_election.cc:302] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:02.551632 5555 raft_consensus.cc:2781] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:02.551723 5555 raft_consensus.cc:482] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:02.551824 5555 raft_consensus.cc:3037] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.553588 4025 catalog_manager.cc:5484] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:02.554330 5555 raft_consensus.cc:504] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.554996 5555 leader_election.cc:288] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:02.555397 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e01586b0ba42869481577ceacc84e8" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:02.555601 4184 raft_consensus.cc:3037] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.555603 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e01586b0ba42869481577ceacc84e8" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:02.555835 4318 raft_consensus.cc:3037] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.558020 4184 raft_consensus.cc:2454] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:11:02.558420 4513 leader_election.cc:302] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:02.558568 4318 raft_consensus.cc:2454] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 1.
I20240501 20:11:02.558704 5555 raft_consensus.cc:2781] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:02.558887 5555 raft_consensus.cc:686] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Becoming Leader. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Running, Role: LEADER
I20240501 20:11:02.559186 5555 consensus_queue.cc:237] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } }
I20240501 20:11:02.562202 4025 catalog_manager.cc:5484] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:02.621845 5802 raft_consensus.cc:482] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:02.622035 5802 raft_consensus.cc:504] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.622787 5802 leader_election.cc:288] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:02.623121 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b4a011c5944a1db9dbc496e87d5768" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:11:02.623346 4580 raft_consensus.cc:2454] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 0.
I20240501 20:11:02.623376 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b4a011c5944a1db9dbc496e87d5768" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:11:02.623617 4184 raft_consensus.cc:2454] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 0.
I20240501 20:11:02.623792 4384 leader_election.cc:302] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:02.624071 5802 raft_consensus.cc:2781] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:02.624217 5802 raft_consensus.cc:482] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:02.624316 5802 raft_consensus.cc:3037] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.626360 5802 raft_consensus.cc:504] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.626935 5802 leader_election.cc:288] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:02.627354 4580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b4a011c5944a1db9dbc496e87d5768" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:11:02.627405 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b4a011c5944a1db9dbc496e87d5768" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:02.627532 4580 raft_consensus.cc:3037] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.627568 4184 raft_consensus.cc:3037] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.629457 4580 raft_consensus.cc:2454] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 1.
I20240501 20:11:02.629485 4184 raft_consensus.cc:2454] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 1.
I20240501 20:11:02.629861 4384 leader_election.cc:302] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:02.630167 5802 raft_consensus.cc:2781] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:02.630563 5802 raft_consensus.cc:686] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Becoming Leader. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Running, Role: LEADER
I20240501 20:11:02.630836 5802 consensus_queue.cc:237] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:02.633879 4025 catalog_manager.cc:5484] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 596d3e24651b4787916397f573c09f1d (127.28.136.67). New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } }
W20240501 20:11:02.981824 5918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:02.982129 5918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:02.989465 5918 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:03.009641 5874 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:03.015498 5874 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:03.021481 5845 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240501 20:11:03.031026 5845 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:03.050437 5802 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:03.057823 5802 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 249.974 ms
time per row: 0.124987 ms
I20240501 20:11:03.412470 4032 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44162:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:03.421054 4428 tablet_service.cc:1467] Processing CreateTablet for tablet e6dc1dbd00224332941bc3301377eb87 (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.421234 4296 tablet_service.cc:1467] Processing CreateTablet for tablet e6dc1dbd00224332941bc3301377eb87 (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.421591 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6dc1dbd00224332941bc3301377eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.421721 4296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6dc1dbd00224332941bc3301377eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.422077 4298 tablet_service.cc:1467] Processing CreateTablet for tablet 11f240637b1e4ef8bc3414e11b38f469 (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.422434 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f240637b1e4ef8bc3414e11b38f469. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.422953 4164 tablet_service.cc:1467] Processing CreateTablet for tablet eeb1eec290ce4dcbb2afedd57e3b81da (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.423352 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb1eec290ce4dcbb2afedd57e3b81da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.424811 4560 tablet_service.cc:1467] Processing CreateTablet for tablet e6dc1dbd00224332941bc3301377eb87 (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.425252 4560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6dc1dbd00224332941bc3301377eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.425732 4297 tablet_service.cc:1467] Processing CreateTablet for tablet eeb1eec290ce4dcbb2afedd57e3b81da (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.426097 4297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb1eec290ce4dcbb2afedd57e3b81da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.429103 4558 tablet_service.cc:1467] Processing CreateTablet for tablet 11f240637b1e4ef8bc3414e11b38f469 (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.429450 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f240637b1e4ef8bc3414e11b38f469. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.433873 4429 tablet_service.cc:1467] Processing CreateTablet for tablet 11f240637b1e4ef8bc3414e11b38f469 (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.433944 4427 tablet_service.cc:1467] Processing CreateTablet for tablet eeb1eec290ce4dcbb2afedd57e3b81da (DEFAULT_TABLE table=rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.434315 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f240637b1e4ef8bc3414e11b38f469. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.437117 4427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb1eec290ce4dcbb2afedd57e3b81da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.451998 5985 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:11:03.454028 5987 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:11:03.454056 5985 tablet_bootstrap.cc:654] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.456663 5987 tablet_bootstrap.cc:654] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.457653 5986 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7: Bootstrap starting.
I20240501 20:11:03.459692 5984 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:11:03.459995 5986 tablet_bootstrap.cc:654] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.460780 5985 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:11:03.461009 5985 ts_tablet_manager.cc:1367] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20240501 20:11:03.461900 5985 raft_consensus.cc:348] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.462244 5985 raft_consensus.cc:374] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.462385 5985 raft_consensus.cc:729] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.462589 5984 tablet_bootstrap.cc:654] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.462787 5985 consensus_queue.cc:260] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.466190 5985 ts_tablet_manager.cc:1398] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:11:03.466542 5985 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79: Bootstrap starting.
I20240501 20:11:03.469254 5985 tablet_bootstrap.cc:654] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.472823 5986 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7: No bootstrap required, opened a new log
I20240501 20:11:03.473047 5986 ts_tablet_manager.cc:1367] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20240501 20:11:03.473522 5987 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:11:03.473757 5987 ts_tablet_manager.cc:1367] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240501 20:11:03.473906 5986 raft_consensus.cc:348] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.474221 5986 raft_consensus.cc:374] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.474357 5986 raft_consensus.cc:729] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.474623 5986 consensus_queue.cc:260] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.474647 5987 raft_consensus.cc:348] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.474874 5987 raft_consensus.cc:374] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.475415 5986 ts_tablet_manager.cc:1398] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:03.475018 5987 raft_consensus.cc:729] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.476351 5987 consensus_queue.cc:260] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.479091 5987 ts_tablet_manager.cc:1398] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:11:03.479498 5987 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:11:03.483494 5802 raft_consensus.cc:482] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:03.483669 5802 raft_consensus.cc:504] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.484463 5802 leader_election.cc:288] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:03.486291 5987 tablet_bootstrap.cc:654] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.488291 5985 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79: No bootstrap required, opened a new log
I20240501 20:11:03.488215 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6dc1dbd00224332941bc3301377eb87" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:11:03.488503 5985 ts_tablet_manager.cc:1367] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.005s
W20240501 20:11:03.489270 4384 leader_election.cc:341] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:03.489349 5985 raft_consensus.cc:348] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.489573 5985 raft_consensus.cc:374] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.489686 5985 raft_consensus.cc:729] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.490038 5985 consensus_queue.cc:260] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.491552 5985 ts_tablet_manager.cc:1398] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:11:03.493443 5987 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:11:03.493667 5987 ts_tablet_manager.cc:1367] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20240501 20:11:03.494496 5987 raft_consensus.cc:348] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.495235 5987 raft_consensus.cc:374] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.495379 5987 raft_consensus.cc:729] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.495412 5984 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:11:03.495602 5984 ts_tablet_manager.cc:1367] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.000s
I20240501 20:11:03.495740 5987 consensus_queue.cc:260] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.496501 5987 ts_tablet_manager.cc:1398] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:03.496419 5984 raft_consensus.cc:348] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.496635 5984 raft_consensus.cc:374] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.496765 5984 raft_consensus.cc:729] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.496815 5987 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: Bootstrap starting.
I20240501 20:11:03.497033 5984 consensus_queue.cc:260] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.497591 5984 ts_tablet_manager.cc:1398] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:03.497862 5984 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:11:03.499002 5987 tablet_bootstrap.cc:654] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.500484 5984 tablet_bootstrap.cc:654] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.501477 5987 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: No bootstrap required, opened a new log
I20240501 20:11:03.501693 5987 ts_tablet_manager.cc:1367] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:11:03.502553 5987 raft_consensus.cc:348] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.502866 5987 raft_consensus.cc:374] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.503029 5987 raft_consensus.cc:729] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.503674 5987 consensus_queue.cc:260] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.504149 5987 ts_tablet_manager.cc:1398] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:03.505381 5984 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:11:03.505532 5984 ts_tablet_manager.cc:1367] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 20:11:03.505895 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6dc1dbd00224332941bc3301377eb87" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d" is_pre_election: true
I20240501 20:11:03.506127 4317 raft_consensus.cc:2454] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 0.
I20240501 20:11:03.506341 5984 raft_consensus.cc:348] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.506628 5984 raft_consensus.cc:374] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.506749 5984 raft_consensus.cc:729] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.506747 4384 leader_election.cc:302] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters: 75b5acf03b9f4f6e98a67ae046587e79
I20240501 20:11:03.507081 5984 consensus_queue.cc:260] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.507206 5802 raft_consensus.cc:2781] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:03.507319 5802 raft_consensus.cc:482] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:03.507421 5802 raft_consensus.cc:3037] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.508277 5984 ts_tablet_manager.cc:1398] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:11:03.508682 5984 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: Bootstrap starting.
I20240501 20:11:03.510026 5802 raft_consensus.cc:504] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.510272 5984 tablet_bootstrap.cc:654] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.510620 5802 leader_election.cc:288] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 election: Requested vote from peers 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:03.511309 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6dc1dbd00224332941bc3301377eb87" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:03.511327 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6dc1dbd00224332941bc3301377eb87" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:11:03.511543 4579 raft_consensus.cc:3037] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.511554 4317 raft_consensus.cc:3037] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.513612 4579 raft_consensus.cc:2454] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 1.
I20240501 20:11:03.514055 4384 leader_election.cc:302] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:03.514633 4317 raft_consensus.cc:2454] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 1.
I20240501 20:11:03.514667 5916 raft_consensus.cc:2781] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:03.514789 5916 raft_consensus.cc:686] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Becoming Leader. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Running, Role: LEADER
I20240501 20:11:03.515120 5916 consensus_queue.cc:237] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } }
I20240501 20:11:03.517742 5984 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: No bootstrap required, opened a new log
I20240501 20:11:03.517879 5984 ts_tablet_manager.cc:1367] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20240501 20:11:03.518457 5984 raft_consensus.cc:348] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.518600 5984 raft_consensus.cc:374] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.518381 4032 catalog_manager.cc:5484] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 596d3e24651b4787916397f573c09f1d (127.28.136.67). New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:03.518682 5984 raft_consensus.cc:729] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.518874 5984 consensus_queue.cc:260] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.519363 5984 ts_tablet_manager.cc:1398] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 20:11:03.569967 5975 raft_consensus.cc:482] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:03.570132 5975 raft_consensus.cc:504] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.570839 5975 leader_election.cc:288] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:03.571199 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f240637b1e4ef8bc3414e11b38f469" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79" is_pre_election: true
I20240501 20:11:03.571424 4579 raft_consensus.cc:2454] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:11:03.571333 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f240637b1e4ef8bc3414e11b38f469" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:11:03.571622 4447 raft_consensus.cc:2454] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:11:03.571799 4253 leader_election.cc:302] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:03.572090 5975 raft_consensus.cc:2781] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:03.572203 5975 raft_consensus.cc:482] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:03.572289 5975 raft_consensus.cc:3037] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.574204 5975 raft_consensus.cc:504] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.574782 5975 leader_election.cc:288] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:03.575124 4579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f240637b1e4ef8bc3414e11b38f469" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:11:03.575212 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11f240637b1e4ef8bc3414e11b38f469" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:11:03.575322 4579 raft_consensus.cc:3037] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.575420 4447 raft_consensus.cc:3037] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.577206 4579 raft_consensus.cc:2454] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:11:03.577307 4447 raft_consensus.cc:2454] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:11:03.577620 4253 leader_election.cc:302] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:03.577930 5975 raft_consensus.cc:2781] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:03.578055 5975 raft_consensus.cc:686] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:11:03.578325 5975 consensus_queue.cc:237] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.581251 4032 catalog_manager.cc:5484] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:03.619148 5862 raft_consensus.cc:482] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:03.619340 5862 raft_consensus.cc:504] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.620136 5862 leader_election.cc:288] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:03.620646 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d" is_pre_election: true
I20240501 20:11:03.620641 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7" is_pre_election: true
I20240501 20:11:03.620887 4447 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:11:03.620906 4184 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 0.
I20240501 20:11:03.621289 4251 leader_election.cc:302] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters:
I20240501 20:11:03.621555 5862 raft_consensus.cc:2781] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:03.621675 5862 raft_consensus.cc:482] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:03.621764 5862 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.623620 5862 raft_consensus.cc:504] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.624136 5862 leader_election.cc:288] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:03.624591 4184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:03.624630 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "4acbc8659ce24119bd77571d1171ee1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:11:03.624847 4447 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.624863 4184 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.627632 4447 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:11:03.627621 4184 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4acbc8659ce24119bd77571d1171ee1d in term 1.
I20240501 20:11:03.628063 4251 leader_election.cc:302] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d; no voters:
I20240501 20:11:03.628369 5862 raft_consensus.cc:2781] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:03.628512 5862 raft_consensus.cc:686] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Leader. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:11:03.628831 5862 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } }
I20240501 20:11:03.632019 4032 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66). New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:04.000466 5862 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:04.006369 5862 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:04.044394 6002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:04.044762 6002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:04.052263 6002 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:04.055287 5916 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:04.063328 5862 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:04.063575 5916 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:04.077028 5975 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
W20240501 20:11:04.310591 4253 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:36125, user_credentials={real_user=slave}} blocked reactor thread for 68741us
INSERT report
rows total: 2000
time total: 240.506 ms
time per row: 0.120253 ms
I20240501 20:11:04.469388 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.69:45155
--local_ip_for_outbound_sockets=127.28.136.69
--tserver_master_addrs=127.28.136.126:33949
--webserver_port=39689
--webserver_interface=127.28.136.69
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240501 20:11:04.820262 6065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:04.820605 6065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:04.820820 6065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:04.829514 6065 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:04.829635 6065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:04.829800 6065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.69
I20240501 20:11:04.839787 6065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.69:45155
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.136.69
--webserver_port=39689
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.69
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:04.841203 6065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:04.843030 6065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:04.855114 6073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:04.855321 6071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:04.855800 6070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:04.856063 6065 server_base.cc:1034] running on GCE node
I20240501 20:11:04.856768 6065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:04.857877 6065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:04.859097 6065 hybrid_clock.cc:648] HybridClock initialized: now 1714594264859029 us; error 52 us; skew 500 ppm
I20240501 20:11:04.859457 6065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:04.863021 6065 webserver.cc:460] Webserver started at http://127.28.136.69:39689/ using document root <none> and password file <none>
I20240501 20:11:04.863924 6065 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:04.864056 6065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:04.870115 6065 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:11:04.873080 6079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:04.874651 6065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:11:04.874796 6065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ee4daecd449e4fc981fd1fe2b0e492c2"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-sx0n"
I20240501 20:11:04.875591 6065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:04.898070 6065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:04.898902 6065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:04.899185 6065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:04.900126 6065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:04.901674 6065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:04.901751 6065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:04.901871 6065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:04.901922 6065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:04.939253 6065 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.69:45155
I20240501 20:11:04.939282 6192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.69:45155 every 8 connection(s)
I20240501 20:11:04.940941 6065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:11:04.947747 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 6065
I20240501 20:11:04.948534 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.70:40983
--local_ip_for_outbound_sockets=127.28.136.70
--tserver_master_addrs=127.28.136.126:33949
--webserver_port=37389
--webserver_interface=127.28.136.70
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:11:04.954335 6193 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:11:04.954638 6193 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:04.955449 6193 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:11:04.957006 4032 ts_manager.cc:194] Re-registered known tserver with Master: ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155)
I20240501 20:11:04.957878 4032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.69:45361
W20240501 20:11:05.301851 6197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:05.302141 6197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:05.302285 6197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:05.310148 6197 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:05.310266 6197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:05.310427 6197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.70
I20240501 20:11:05.323338 6197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.70:40983
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.136.70
--webserver_port=37389
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.70
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:05.325345 6197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:05.327766 6197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:05.340991 6205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:05.340940 6207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:05.341457 6204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:05.342360 6197 server_base.cc:1034] running on GCE node
I20240501 20:11:05.342960 6197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:05.343907 6197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:05.345109 6197 hybrid_clock.cc:648] HybridClock initialized: now 1714594265345055 us; error 45 us; skew 500 ppm
I20240501 20:11:05.345420 6197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:05.347560 6197 webserver.cc:460] Webserver started at http://127.28.136.70:37389/ using document root <none> and password file <none>
I20240501 20:11:05.348304 6197 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:05.348428 6197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:05.353811 6197 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:11:05.356433 6213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.357779 6197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:11:05.357935 6197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f5195730e32a4abd8519baf9961d32f2"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-sx0n"
I20240501 20:11:05.358704 6197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:05.379460 6197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:05.380304 6197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:05.380625 6197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:05.381645 6197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:05.383176 6197 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:05.383255 6197 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.383371 6197 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:05.383423 6197 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.420296 6197 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.70:40983
I20240501 20:11:05.420348 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.70:40983 every 8 connection(s)
I20240501 20:11:05.421990 6197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:11:05.425792 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 6197
I20240501 20:11:05.426369 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.71:40971
--local_ip_for_outbound_sockets=127.28.136.71
--tserver_master_addrs=127.28.136.126:33949
--webserver_port=33559
--webserver_interface=127.28.136.71
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:11:05.436133 6329 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:11:05.436488 6329 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:05.437533 6329 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:11:05.439723 4032 ts_manager.cc:194] Re-registered known tserver with Master: f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983)
I20240501 20:11:05.440865 4032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.70:47051
W20240501 20:11:05.782492 6332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:05.782774 6332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:05.782905 6332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:05.790457 6332 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:05.790568 6332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:05.790730 6332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.71
I20240501 20:11:05.800055 6332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.71:40971
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.136.71
--webserver_port=33559
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.71
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:05.801779 6332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:05.803534 6332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:05.816944 6339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:05.817330 6341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:05.817781 6338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:05.818111 6332 server_base.cc:1034] running on GCE node
I20240501 20:11:05.818701 6332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:05.819646 6332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:05.820837 6332 hybrid_clock.cc:648] HybridClock initialized: now 1714594265820776 us; error 42 us; skew 500 ppm
I20240501 20:11:05.821220 6332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:05.823594 6332 webserver.cc:460] Webserver started at http://127.28.136.71:33559/ using document root <none> and password file <none>
I20240501 20:11:05.824344 6332 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:05.824476 6332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:05.830260 6332 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 20:11:05.833220 6347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.834625 6332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:11:05.834794 6332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9592068f16ca406eb8a347a65d365782"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-sx0n"
I20240501 20:11:05.835664 6332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:05.863137 6332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:05.863992 6332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:05.864264 6332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:05.865253 6332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:05.867444 6332 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:05.867522 6332 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.867648 6332 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:05.867698 6332 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.906796 6332 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.71:40971
I20240501 20:11:05.906821 6461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.71:40971 every 8 connection(s)
I20240501 20:11:05.908244 6332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:11:05.921059 6462 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:11:05.921409 6462 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:05.922273 6462 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:11:05.923849 4032 ts_manager.cc:194] Re-registered known tserver with Master: 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971)
I20240501 20:11:05.924700 4032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.71:54125
I20240501 20:11:05.950608 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 6332
I20240501 20:11:05.951156 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.72:46003
--local_ip_for_outbound_sockets=127.28.136.72
--tserver_master_addrs=127.28.136.126:33949
--webserver_port=34873
--webserver_interface=127.28.136.72
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:11:05.960394 6193 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
W20240501 20:11:06.303184 6465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:06.303495 6465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:06.303637 6465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:06.311313 6465 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:06.311446 6465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:06.311601 6465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.72
I20240501 20:11:06.321193 6465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.72:46003
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.136.72
--webserver_port=34873
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.72
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:06.322520 6465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:06.324559 6465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:06.336694 6471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:06.337342 6465 server_base.cc:1034] running on GCE node
W20240501 20:11:06.337801 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:06.337913 6473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:06.338596 6465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:06.339625 6465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:06.340829 6465 hybrid_clock.cc:648] HybridClock initialized: now 1714594266340766 us; error 59 us; skew 500 ppm
I20240501 20:11:06.341168 6465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:06.343581 6465 webserver.cc:460] Webserver started at http://127.28.136.72:34873/ using document root <none> and password file <none>
I20240501 20:11:06.344342 6465 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:06.344480 6465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:06.350087 6465 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 20:11:06.352778 6479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.354101 6465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:06.354236 6465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "55fe2a4464a9402f927b2b60609b34dd"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-sx0n"
I20240501 20:11:06.355006 6465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:06.387480 6465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:06.388331 6465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:06.388626 6465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:06.389642 6465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:06.391108 6465 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:06.391196 6465 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.391333 6465 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:06.391383 6465 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.421044 6465 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.72:46003
I20240501 20:11:06.421135 6592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.72:46003 every 8 connection(s)
I20240501 20:11:06.422489 6465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:11:06.427858 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 6465
I20240501 20:11:06.444154 6329 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
I20240501 20:11:06.452780 6593 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:11:06.453122 6593 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:06.454185 6593 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:11:06.455888 4032 ts_manager.cc:194] Re-registered known tserver with Master: 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003)
I20240501 20:11:06.456964 4032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.72:35369
W20240501 20:11:06.803728 6599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:06.804072 6599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:06.815276 6599 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:06.927214 6462 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
I20240501 20:11:07.066118 4296 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:07.085824 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.099632 4296 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:07.115514 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.138741 4428 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:07.161355 4428 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:07.190191 4164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:07.190835 4164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:07.228087 4558 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:07.289338 4558 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:07.303367 6128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.324191 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.331081 6128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.335202 6261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.361199 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.378790 6261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.460106 6593 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
I20240501 20:11:07.675288 4296 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:07.707300 4428 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:07.761205 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.773993 6261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.776340 4164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:07.776648 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.780275 4558 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:07.796882 6128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.071579 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.072199 4296 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:08.086277 4428 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:08.096868 4164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:08.131026 4558 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 20:11:08.141516 4515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:38397, user_credentials={real_user=slave}} blocked reactor thread for 54841us
I20240501 20:11:08.153201 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.170867 6128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.181792 6261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 20:11:08.228205 6729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:08.228610 6729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:08.238303 6729 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:08.288339 4317 consensus_queue.cc:237] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:08.291525 4579 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.292487 4184 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.292985 6696 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.294525 6788 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:08.300186 4251 consensus_peers.cc:485] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0822a3fc82134d018bae96fecc62fdd3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:08.307281 6787 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:08.308070 4184 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:08.311513 4579 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:08.317509 4032 catalog_manager.cc:5484] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.358652 4186 consensus_queue.cc:237] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:08.361675 4447 raft_consensus.cc:1261] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.362705 6202 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.362877 4317 raft_consensus.cc:1261] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.363566 6202 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:08.388814 4116 consensus_peers.cc:485] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a452749c5c44b87bb495983853cea3a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:08.389231 6202 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:08.393922 4318 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:08.394335 4449 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:08.401031 4030 catalog_manager.cc:5484] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:08.431730 4317 consensus_queue.cc:237] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:08.438367 4447 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.438952 6788 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.439525 4579 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.440939 6788 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.443899 6788 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:08.444372 4579 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
W20240501 20:11:08.449335 4252 consensus_peers.cc:485] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ff47913ab3b4771abbd9c912bbf34ad. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:08.449985 4032 catalog_manager.cc:5484] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.484099 4449 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:08.529222 4447 consensus_queue.cc:237] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:08.537853 4579 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.541530 5916 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240501 20:11:08.549214 4186 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.550119 6859 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.555106 6859 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:08.559000 4296 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:08.559973 4579 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:08.563241 4032 catalog_manager.cc:5484] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.579066 4185 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
W20240501 20:11:08.583246 4382 consensus_peers.cc:485] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb36776ab8b74586aead4ecad56cc815. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:08.604357 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.618225 4428 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:08.622172 6261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.638186 4558 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:08.645076 4164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:08.653069 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.668614 6128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.696290 4185 consensus_queue.cc:237] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:08.699309 4447 raft_consensus.cc:1261] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.699324 4580 raft_consensus.cc:1261] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:08.699916 4116 consensus_peers.cc:485] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1564f07420a346dbaa77ef6a2a1a1b42. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:08.699991 6202 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.700304 6600 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.726027 6835 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:08.733470 4580 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:08.735245 4032 catalog_manager.cc:5484] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.736951 4447 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
Runtime error: ksck discovered errors
I20240501 20:11:08.765349 4317 consensus_queue.cc:237] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:08.767880 4447 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:08.768055 4251 consensus_peers.cc:485] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbd253f3e7824a81b6fdaf2c8880aaa9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:08.768472 6787 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.768540 4185 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.769271 6787 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.771219 6788 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:08.771790 4447 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:08.772696 4185 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:08.777561 4030 catalog_manager.cc:5484] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:08.784503 6892 ts_tablet_manager.cc:902] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:08.795286 4317 consensus_queue.cc:237] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:08.797797 4580 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.798631 4447 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.799224 6787 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:08.799786 4252 consensus_peers.cc:485] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e97fe7ab97a4458b5e977af3d1356df. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:08.800727 6787 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.806021 6787 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:08.809543 4447 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:08.813226 6892 tablet_copy_client.cc:308] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:08.818977 4030 catalog_manager.cc:5484] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:08.821449 4579 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:08.844388 6904 ts_tablet_manager.cc:902] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:08.851857 6906 ts_tablet_manager.cc:902] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:08.859915 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 0a452749c5c44b87bb495983853cea3a from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:42689)
I20240501 20:11:08.860179 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 0a452749c5c44b87bb495983853cea3a from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:42689: session id = 55fe2a4464a9402f927b2b60609b34dd-0a452749c5c44b87bb495983853cea3a
I20240501 20:11:08.863898 4206 tablet_copy_source_session.cc:205] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:08.869804 6892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a452749c5c44b87bb495983853cea3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:08.880378 6906 tablet_copy_client.cc:308] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:08.889303 6892 tablet_copy_client.cc:783] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:08.889963 6892 tablet_copy_client.cc:647] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:08.893081 6900 tablet_copy_client.cc:1002] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:08.903652 4317 consensus_queue.cc:237] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:08.907387 4580 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:08.908048 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 0822a3fc82134d018bae96fecc62fdd3 from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:42579)
I20240501 20:11:08.908241 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 0822a3fc82134d018bae96fecc62fdd3 from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:42579: session id = ee4daecd449e4fc981fd1fe2b0e492c2-0822a3fc82134d018bae96fecc62fdd3
I20240501 20:11:08.911490 6787 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.912369 4185 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:08.912799 4252 consensus_peers.cc:485] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b344760b82cb4bc09379d990f9558121. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:08.912981 6787 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.915362 4338 tablet_copy_source_session.cc:205] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:08.917179 6904 tablet_copy_client.cc:308] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:08.917922 6787 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:08.921272 6906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0822a3fc82134d018bae96fecc62fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:08.945111 4580 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:08.946466 6906 tablet_copy_client.cc:783] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:08.946767 6906 tablet_copy_client.cc:647] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:08.949023 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 9ff47913ab3b4771abbd9c912bbf34ad from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:48699)
I20240501 20:11:08.949213 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 9ff47913ab3b4771abbd9c912bbf34ad from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:48699: session id = f5195730e32a4abd8519baf9961d32f2-9ff47913ab3b4771abbd9c912bbf34ad
I20240501 20:11:08.949265 6915 tablet_copy_client.cc:1002] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:08.951951 4338 tablet_copy_source_session.cc:205] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:08.954852 4030 catalog_manager.cc:5484] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.955456 6933 ts_tablet_manager.cc:902] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:08.961774 6904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff47913ab3b4771abbd9c912bbf34ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.009420 4185 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:09.045218 6933 tablet_copy_client.cc:308] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38865
I20240501 20:11:09.048316 6904 tablet_copy_client.cc:783] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.052235 6904 tablet_copy_client.cc:647] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.053201 4469 tablet_copy_service.cc:140] P 596d3e24651b4787916397f573c09f1d: Received BeginTabletCopySession request for tablet bb36776ab8b74586aead4ecad56cc815 from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:43141)
I20240501 20:11:09.053436 4469 tablet_copy_service.cc:161] P 596d3e24651b4787916397f573c09f1d: Beginning new tablet copy session on tablet bb36776ab8b74586aead4ecad56cc815 from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:43141: session id = 9592068f16ca406eb8a347a65d365782-bb36776ab8b74586aead4ecad56cc815
I20240501 20:11:09.055246 6917 tablet_copy_client.cc:1002] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.056301 4469 tablet_copy_source_session.cc:205] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.058508 4317 consensus_queue.cc:237] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:09.062601 6933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb36776ab8b74586aead4ecad56cc815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.070118 4580 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.070137 4447 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.070792 6927 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.071537 6696 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:09.078073 4251 consensus_peers.cc:485] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79cb3f8b3fe94497bb54d3d8b7f18754. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.078642 6933 tablet_copy_client.cc:783] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.079072 6933 tablet_copy_client.cc:647] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.081725 6938 tablet_copy_client.cc:1002] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.086670 6696 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.087409 4579 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.094530 4032 catalog_manager.cc:5484] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.100548 4447 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.117368 4185 consensus_queue.cc:237] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:09.120193 4317 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.121623 4447 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.122148 6835 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.122499 6202 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:09.131734 4115 consensus_peers.cc:485] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d09b0474ecbf4d458d7a520c9e9f329b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.138940 6202 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.140007 4317 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.143635 4447 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.145386 4026 catalog_manager.cc:5484] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:09.161623 4185 consensus_queue.cc:237] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:09.164573 4447 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.164669 4317 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.165494 6835 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.166200 6600 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.172048 6835 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
W20240501 20:11:09.173049 4115 consensus_peers.cc:485] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60871b6059c249bfaac713e2f7325811. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.174547 4447 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.180475 4032 catalog_manager.cc:5484] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.184073 4318 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.205399 4447 consensus_queue.cc:237] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:09.207708 4580 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.208395 4185 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.208339 5916 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.209316 5916 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.231616 5916 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:09.236596 4580 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:09.237814 4032 catalog_manager.cc:5484] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:09.249006 4185 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:09.256012 6971 ts_tablet_manager.cc:902] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
W20240501 20:11:09.263124 4383 consensus_peers.cc:485] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b4a011c5944a1db9dbc496e87d5768. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.272684 6971 tablet_copy_client.cc:308] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:09.273521 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 1564f07420a346dbaa77ef6a2a1a1b42 from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:42689)
I20240501 20:11:09.273703 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 1564f07420a346dbaa77ef6a2a1a1b42 from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:42689: session id = 55fe2a4464a9402f927b2b60609b34dd-1564f07420a346dbaa77ef6a2a1a1b42
I20240501 20:11:09.275633 4206 tablet_copy_source_session.cc:205] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.277025 6971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1564f07420a346dbaa77ef6a2a1a1b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.284816 6971 tablet_copy_client.cc:783] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.285166 6971 tablet_copy_client.cc:647] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.286813 6978 tablet_copy_client.cc:1002] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.287254 4317 consensus_queue.cc:237] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:09.289884 4580 raft_consensus.cc:1261] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:09.289917 4252 consensus_peers.cc:485] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28b9b57213a64172afdc2ca890569fc5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.290416 6787 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.290745 4447 raft_consensus.cc:1261] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.291568 6787 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.295243 6696 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:09.296250 4580 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:09.300150 4449 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:09.301359 4030 catalog_manager.cc:5484] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:09.314071 6989 ts_tablet_manager.cc:902] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:09.321369 6989 tablet_copy_client.cc:308] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:09.323704 4185 consensus_queue.cc:237] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
W20240501 20:11:09.330081 4115 consensus_peers.cc:485] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96005b2bfa9847bb992640ad40a68b36. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.330772 4449 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.331421 6600 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.331521 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet b344760b82cb4bc09379d990f9558121 from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:50629)
I20240501 20:11:09.331662 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet b344760b82cb4bc09379d990f9558121 from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:50629: session id = 55fe2a4464a9402f927b2b60609b34dd-b344760b82cb4bc09379d990f9558121
I20240501 20:11:09.333472 4317 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.334571 4338 tablet_copy_source_session.cc:205] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.335016 6600 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.337258 6989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b344760b82cb4bc09379d990f9558121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.344590 6989 tablet_copy_client.cc:783] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.344923 6989 tablet_copy_client.cc:647] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.346717 6993 tablet_copy_client.cc:1002] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.350420 6600 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.353255 6997 ts_tablet_manager.cc:902] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:09.355912 4449 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.358983 4030 catalog_manager.cc:5484] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:09.382232 4317 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.394737 7008 ts_tablet_manager.cc:902] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:09.411526 4580 consensus_queue.cc:237] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:09.417398 6997 tablet_copy_client.cc:308] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:09.418666 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 2e97fe7ab97a4458b5e977af3d1356df from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:48699)
I20240501 20:11:09.418854 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 2e97fe7ab97a4458b5e977af3d1356df from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:48699: session id = f5195730e32a4abd8519baf9961d32f2-2e97fe7ab97a4458b5e977af3d1356df
I20240501 20:11:09.427260 4338 tablet_copy_source_session.cc:205] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.442844 6997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e97fe7ab97a4458b5e977af3d1356df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.454946 4449 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.456066 6785 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.459789 6785 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.465404 4449 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.472455 4030 catalog_manager.cc:5484] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.488268 4317 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.488998 7019 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:09.496481 4513 consensus_peers.cc:485] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d48a2b58ce2d417e9abf63c292419d14. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.496886 6997 tablet_copy_client.cc:783] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.497228 6997 tablet_copy_client.cc:647] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.499315 7010 tablet_copy_client.cc:1002] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.507755 4317 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.518967 7028 ts_tablet_manager.cc:902] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:09.520236 7008 tablet_copy_client.cc:308] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:09.521126 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet bbd253f3e7824a81b6fdaf2c8880aaa9 from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:42579)
W20240501 20:11:09.520756 6935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240501 20:11:09.521274 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet bbd253f3e7824a81b6fdaf2c8880aaa9 from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:42579: session id = ee4daecd449e4fc981fd1fe2b0e492c2-bbd253f3e7824a81b6fdaf2c8880aaa9
W20240501 20:11:09.521374 6935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:11:09.524873 4338 tablet_copy_source_session.cc:205] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.525116 4185 consensus_queue.cc:237] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:09.527197 7008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd253f3e7824a81b6fdaf2c8880aaa9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:11:09.533485 6935 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:09.533890 7008 tablet_copy_client.cc:783] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.534190 7008 tablet_copy_client.cc:647] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.539126 4580 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.541220 6835 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.542929 4317 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.543850 6835 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:09.545852 4116 consensus_peers.cc:485] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f7fb3f245414d05a8e5d837076195c2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.549664 6202 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:09.550935 7028 tablet_copy_client.cc:308] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:09.553913 4579 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:09.559840 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:52891)
I20240501 20:11:09.560015 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 79cb3f8b3fe94497bb54d3d8b7f18754 from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:52891: session id = 9592068f16ca406eb8a347a65d365782-79cb3f8b3fe94497bb54d3d8b7f18754
I20240501 20:11:09.560225 4030 catalog_manager.cc:5484] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:09.563181 4338 tablet_copy_source_session.cc:205] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.564579 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79cb3f8b3fe94497bb54d3d8b7f18754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.566124 7029 tablet_copy_client.cc:1002] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.570758 7028 tablet_copy_client.cc:783] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.571105 7028 tablet_copy_client.cc:647] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.572687 7038 tablet_copy_client.cc:1002] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.573582 4318 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:09.596958 4185 consensus_queue.cc:237] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:09.599148 4449 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:09.599552 4115 consensus_peers.cc:485] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d7b1a47b1fe4e4b9a60b18b0b99056b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.600040 6202 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.600625 4579 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.601605 6202 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.602685 6600 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.603328 4449 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.608924 4030 catalog_manager.cc:5484] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.624094 4579 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:09.633920 4579 consensus_queue.cc:237] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:09.636831 4185 raft_consensus.cc:1261] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.637598 4318 raft_consensus.cc:1261] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.638154 6785 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.639261 7020 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:09.649993 4514 consensus_peers.cc:485] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbf736b8dba9474da5fb7ca9d9348ac7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.650722 7019 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:09.653673 7059 ts_tablet_manager.cc:902] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:09.655815 4186 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:09.663141 4032 catalog_manager.cc:5484] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:09.677896 4317 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:09.687232 7072 ts_tablet_manager.cc:902] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:09.698737 7072 tablet_copy_client.cc:308] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:09.699640 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 28b9b57213a64172afdc2ca890569fc5 from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:50629)
I20240501 20:11:09.699813 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 28b9b57213a64172afdc2ca890569fc5 from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:50629: session id = 55fe2a4464a9402f927b2b60609b34dd-28b9b57213a64172afdc2ca890569fc5
I20240501 20:11:09.702853 4338 tablet_copy_source_session.cc:205] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.703256 7059 tablet_copy_client.cc:308] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:09.706404 7072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b9b57213a64172afdc2ca890569fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.712488 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 60871b6059c249bfaac713e2f7325811 from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:50981)
I20240501 20:11:09.712661 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 60871b6059c249bfaac713e2f7325811 from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:50981: session id = 9592068f16ca406eb8a347a65d365782-60871b6059c249bfaac713e2f7325811
I20240501 20:11:09.713065 7072 tablet_copy_client.cc:783] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.713330 7072 tablet_copy_client.cc:647] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.715313 4206 tablet_copy_source_session.cc:205] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.716660 7059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60871b6059c249bfaac713e2f7325811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.716807 7083 tablet_copy_client.cc:1002] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.718145 4317 consensus_queue.cc:237] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:09.720959 4449 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:09.720959 4579 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:09.721356 4251 consensus_peers.cc:485] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cdac8c0b92d442ebd84fb43af7e32fb. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:09.721530 6787 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.722837 7059 tablet_copy_client.cc:783] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.723104 7059 tablet_copy_client.cc:647] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.723440 6787 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:09.725447 7082 tablet_copy_client.cc:1002] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.740880 7090 ts_tablet_manager.cc:902] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:09.745707 7090 tablet_copy_client.cc:308] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:09.750849 6787 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.753501 4580 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.760232 7100 ts_tablet_manager.cc:902] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:09.763417 4032 catalog_manager.cc:5484] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.764380 4449 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:09.767138 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet d09b0474ecbf4d458d7a520c9e9f329b from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:60381)
I20240501 20:11:09.767308 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet d09b0474ecbf4d458d7a520c9e9f329b from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:60381: session id = ee4daecd449e4fc981fd1fe2b0e492c2-d09b0474ecbf4d458d7a520c9e9f329b
I20240501 20:11:09.770291 4206 tablet_copy_source_session.cc:205] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.771962 7090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09b0474ecbf4d458d7a520c9e9f329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.776592 7093 ts_tablet_manager.cc:902] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:09.780967 7090 tablet_copy_client.cc:783] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.781356 7090 tablet_copy_client.cc:647] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.783223 7097 tablet_copy_client.cc:1002] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.809212 7100 tablet_copy_client.cc:308] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:09.810052 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 96005b2bfa9847bb992640ad40a68b36 from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:60381)
I20240501 20:11:09.810245 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 96005b2bfa9847bb992640ad40a68b36 from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:60381: session id = ee4daecd449e4fc981fd1fe2b0e492c2-96005b2bfa9847bb992640ad40a68b36
I20240501 20:11:09.813519 4206 tablet_copy_source_session.cc:205] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.821857 7100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96005b2bfa9847bb992640ad40a68b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.829552 7100 tablet_copy_client.cc:783] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.829891 7100 tablet_copy_client.cc:647] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.831686 7112 tablet_copy_client.cc:1002] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.854612 7117 ts_tablet_manager.cc:902] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:09.874465 7093 tablet_copy_client.cc:308] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38865
I20240501 20:11:09.906812 7117 tablet_copy_client.cc:308] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36125
I20240501 20:11:09.924679 4469 tablet_copy_service.cc:140] P 596d3e24651b4787916397f573c09f1d: Received BeginTabletCopySession request for tablet c3b4a011c5944a1db9dbc496e87d5768 from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:58253)
I20240501 20:11:09.924925 4469 tablet_copy_service.cc:161] P 596d3e24651b4787916397f573c09f1d: Beginning new tablet copy session on tablet c3b4a011c5944a1db9dbc496e87d5768 from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:58253: session id = f5195730e32a4abd8519baf9961d32f2-c3b4a011c5944a1db9dbc496e87d5768
I20240501 20:11:09.927896 4469 tablet_copy_source_session.cc:205] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.928022 4600 tablet_copy_service.cc:140] P 75b5acf03b9f4f6e98a67ae046587e79: Received BeginTabletCopySession request for tablet d48a2b58ce2d417e9abf63c292419d14 from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:44331)
I20240501 20:11:09.928297 4600 tablet_copy_service.cc:161] P 75b5acf03b9f4f6e98a67ae046587e79: Beginning new tablet copy session on tablet d48a2b58ce2d417e9abf63c292419d14 from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:44331: session id = 9592068f16ca406eb8a347a65d365782-d48a2b58ce2d417e9abf63c292419d14
I20240501 20:11:09.932351 4600 tablet_copy_source_session.cc:205] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:09.934826 7093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b4a011c5944a1db9dbc496e87d5768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.940513 7117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48a2b58ce2d417e9abf63c292419d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:09.949566 7117 tablet_copy_client.cc:783] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.950688 7117 tablet_copy_client.cc:647] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.950986 7093 tablet_copy_client.cc:783] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:09.951814 7093 tablet_copy_client.cc:647] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:09.953460 7124 tablet_copy_client.cc:1002] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:09.953847 7115 tablet_copy_client.cc:1002] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:10.003526 7133 ts_tablet_manager.cc:902] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:10.007464 4296 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:10.027719 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.069249 4558 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:11:10.073446 4428 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:10.100781 6128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.118366 6261 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.120841 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:10.138388 4164 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:11:10.170791 7133 tablet_copy_client.cc:308] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:10.195348 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 3f7fb3f245414d05a8e5d837076195c2 from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:39057)
I20240501 20:11:10.195561 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 3f7fb3f245414d05a8e5d837076195c2 from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:39057: session id = f5195730e32a4abd8519baf9961d32f2-3f7fb3f245414d05a8e5d837076195c2
I20240501 20:11:10.198747 4206 tablet_copy_source_session.cc:205] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:10.200970 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7fb3f245414d05a8e5d837076195c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:10.207407 7133 tablet_copy_client.cc:783] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:10.207803 7133 tablet_copy_client.cc:647] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:10.210096 7161 tablet_copy_client.cc:1002] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:10.234411 7163 ts_tablet_manager.cc:902] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:10.245206 7163 tablet_copy_client.cc:308] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:10.246140 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:50981)
I20240501 20:11:10.246356 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:50981: session id = 9592068f16ca406eb8a347a65d365782-3d7b1a47b1fe4e4b9a60b18b0b99056b
I20240501 20:11:10.249738 4206 tablet_copy_source_session.cc:205] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:10.251783 7173 ts_tablet_manager.cc:902] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:10.255630 7163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:10.264010 7172 ts_tablet_manager.cc:902] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:10.267479 7163 tablet_copy_client.cc:783] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:10.267778 7163 tablet_copy_client.cc:647] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:10.269582 7171 tablet_copy_client.cc:1002] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:10.277508 7173 tablet_copy_client.cc:308] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36125
I20240501 20:11:10.278421 7172 tablet_copy_client.cc:308] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:10.279217 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 5cdac8c0b92d442ebd84fb43af7e32fb from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:42579)
I20240501 20:11:10.279413 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 5cdac8c0b92d442ebd84fb43af7e32fb from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:42579: session id = ee4daecd449e4fc981fd1fe2b0e492c2-5cdac8c0b92d442ebd84fb43af7e32fb
I20240501 20:11:10.282588 4338 tablet_copy_source_session.cc:205] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:10.284663 4600 tablet_copy_service.cc:140] P 75b5acf03b9f4f6e98a67ae046587e79: Received BeginTabletCopySession request for tablet bbf736b8dba9474da5fb7ca9d9348ac7 from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:48463)
I20240501 20:11:10.284902 4600 tablet_copy_service.cc:161] P 75b5acf03b9f4f6e98a67ae046587e79: Beginning new tablet copy session on tablet bbf736b8dba9474da5fb7ca9d9348ac7 from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:48463: session id = 55fe2a4464a9402f927b2b60609b34dd-bbf736b8dba9474da5fb7ca9d9348ac7
I20240501 20:11:10.285315 7172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdac8c0b92d442ebd84fb43af7e32fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:10.287446 4600 tablet_copy_source_session.cc:205] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:10.290627 7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbf736b8dba9474da5fb7ca9d9348ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:10.291965 7172 tablet_copy_client.cc:783] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:10.292207 7172 tablet_copy_client.cc:647] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:10.295675 7186 tablet_copy_client.cc:1002] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
Runtime error: ksck discovered errors
I20240501 20:11:10.299211 7173 tablet_copy_client.cc:783] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:10.299420 7173 tablet_copy_client.cc:647] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:10.300904 7185 tablet_copy_client.cc:1002] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:10.394227 6892 tablet_copy_client.cc:523] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:10.399149 6892 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:10.423153 6892 log.cc:824] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:10.426839 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.73:0
--local_ip_for_outbound_sockets=127.28.136.73
--webserver_interface=127.28.136.73
--webserver_port=0
--tserver_master_addrs=127.28.136.126:33949
--builtin_ntp_servers=127.28.136.84:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240501 20:11:10.449802 6906 tablet_copy_client.cc:523] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:10.455119 6906 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:10.475992 6892 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:10.476709 6906 log.cc:824] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:10.476819 6892 tablet_bootstrap.cc:492] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:10.477365 6892 ts_tablet_manager.cc:1367] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.078s user 0.058s sys 0.005s
I20240501 20:11:10.484035 6892 raft_consensus.cc:348] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:10.484719 6892 raft_consensus.cc:729] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:10.485476 6892 consensus_queue.cc:260] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:10.504942 6892 ts_tablet_manager.cc:1398] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.027s user 0.010s sys 0.016s
I20240501 20:11:10.506393 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-0a452749c5c44b87bb495983853cea3a received from {username='slave'} at 127.28.136.72:42689
I20240501 20:11:10.506590 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-0a452749c5c44b87bb495983853cea3a on tablet 0a452749c5c44b87bb495983853cea3a with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:10.526774 6906 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:10.527385 6906 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:10.527905 6906 ts_tablet_manager.cc:1367] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.073s user 0.052s sys 0.008s
I20240501 20:11:10.533851 6906 raft_consensus.cc:348] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:10.534409 6906 raft_consensus.cc:729] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:10.534989 6906 consensus_queue.cc:260] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:10.543301 6906 ts_tablet_manager.cc:1398] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
I20240501 20:11:10.544332 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-0822a3fc82134d018bae96fecc62fdd3 received from {username='slave'} at 127.28.136.69:42579
I20240501 20:11:10.544492 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-0822a3fc82134d018bae96fecc62fdd3 on tablet 0822a3fc82134d018bae96fecc62fdd3 with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:10.555819 6904 tablet_copy_client.cc:523] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:10.560284 6904 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:10.572340 6904 log.cc:824] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:10.582247 6933 tablet_copy_client.cc:523] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:10.586699 6933 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:10.607663 6933 log.cc:824] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:10.621955 6904 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:10.622658 6904 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:10.623232 6904 ts_tablet_manager.cc:1367] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.063s user 0.059s sys 0.000s
I20240501 20:11:10.638235 6904 raft_consensus.cc:348] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:10.638887 6904 raft_consensus.cc:729] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:10.639518 6904 consensus_queue.cc:260] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:10.644188 6904 ts_tablet_manager.cc:1398] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.021s user 0.008s sys 0.004s
I20240501 20:11:10.648448 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-9ff47913ab3b4771abbd9c912bbf34ad received from {username='slave'} at 127.28.136.70:48699
I20240501 20:11:10.648597 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-9ff47913ab3b4771abbd9c912bbf34ad on tablet 9ff47913ab3b4771abbd9c912bbf34ad with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:10.660785 6933 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:10.661304 6933 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:10.661710 6933 ts_tablet_manager.cc:1367] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.075s user 0.056s sys 0.005s
I20240501 20:11:10.667673 6282 raft_consensus.cc:1203] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:10.667712 6933 raft_consensus.cc:348] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:10.668197 6933 raft_consensus.cc:729] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:10.668856 6933 consensus_queue.cc:260] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:10.671212 6933 ts_tablet_manager.cc:1398] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 20:11:10.672446 4469 tablet_copy_service.cc:342] P 596d3e24651b4787916397f573c09f1d: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-bb36776ab8b74586aead4ecad56cc815 received from {username='slave'} at 127.28.136.71:43141
I20240501 20:11:10.672602 4469 tablet_copy_service.cc:433] P 596d3e24651b4787916397f573c09f1d: ending tablet copy session 9592068f16ca406eb8a347a65d365782-bb36776ab8b74586aead4ecad56cc815 on tablet bb36776ab8b74586aead4ecad56cc815 with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:10.787220 6971 tablet_copy_client.cc:523] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:10.790743 6971 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:10.820318 6148 raft_consensus.cc:1203] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:10.827909 6971 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:10.828441 6971 tablet_bootstrap.cc:492] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:10.828778 6971 ts_tablet_manager.cc:1367] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.038s user 0.035s sys 0.003s
I20240501 20:11:10.829777 6971 raft_consensus.cc:348] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:10.830075 6971 raft_consensus.cc:729] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:10.830282 6971 consensus_queue.cc:260] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:10.830960 6971 ts_tablet_manager.cc:1398] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:10.832006 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-1564f07420a346dbaa77ef6a2a1a1b42 received from {username='slave'} at 127.28.136.72:42689
I20240501 20:11:10.832170 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-1564f07420a346dbaa77ef6a2a1a1b42 on tablet 1564f07420a346dbaa77ef6a2a1a1b42 with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:10.847083 6989 tablet_copy_client.cc:523] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:10.850750 6989 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:10.917834 6547 raft_consensus.cc:1203] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:10.933706 6989 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:10.934350 6989 tablet_bootstrap.cc:492] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:10.934737 6989 ts_tablet_manager.cc:1367] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.084s user 0.050s sys 0.005s
I20240501 20:11:10.935647 6989 raft_consensus.cc:348] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:10.936069 6989 raft_consensus.cc:729] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:10.936327 6989 consensus_queue.cc:260] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:10.937186 6989 ts_tablet_manager.cc:1398] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:10.938370 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-b344760b82cb4bc09379d990f9558121 received from {username='slave'} at 127.28.136.72:50629
I20240501 20:11:10.938501 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-b344760b82cb4bc09379d990f9558121 on tablet b344760b82cb4bc09379d990f9558121 with peer 55fe2a4464a9402f927b2b60609b34dd
W20240501 20:11:10.947702 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:10.948092 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:10.948288 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:10.959748 7234 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:10.959947 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:10.960141 7234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.73
I20240501 20:11:10.972664 7234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.73:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.28.136.73
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.28.136.126:33949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.73
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:10.974627 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:10.976923 7234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:10.992022 7304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:10.992460 7302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:10.993700 7306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:10.994586 7234 server_base.cc:1034] running on GCE node
I20240501 20:11:10.995286 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:10.996886 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:10.998127 7234 hybrid_clock.cc:648] HybridClock initialized: now 1714594270998069 us; error 51 us; skew 500 ppm
I20240501 20:11:10.998507 7234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:10.999686 6997 tablet_copy_client.cc:523] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.003247 6997 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:11.013654 7234 webserver.cc:460] Webserver started at http://127.28.136.73:43171/ using document root <none> and password file <none>
I20240501 20:11:11.014839 7234 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:11.015607 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:11.016003 7234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:11.016489 6416 raft_consensus.cc:1203] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.019862 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "b662c3da2b1b4f3693920ccb680b8d12"
format_stamp: "Formatted at 2024-05-01 20:11:11 on dist-test-slave-sx0n"
I20240501 20:11:11.021689 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "b662c3da2b1b4f3693920ccb680b8d12"
format_stamp: "Formatted at 2024-05-01 20:11:11 on dist-test-slave-sx0n"
I20240501 20:11:11.029083 7234 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240501 20:11:11.037377 7327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:11.039383 7234 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.003s
I20240501 20:11:11.039557 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "b662c3da2b1b4f3693920ccb680b8d12"
format_stamp: "Formatted at 2024-05-01 20:11:11 on dist-test-slave-sx0n"
I20240501 20:11:11.039808 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:11.051746 6997 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.052258 6997 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:11.052560 6997 ts_tablet_manager.cc:1367] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.049s user 0.040s sys 0.004s
I20240501 20:11:11.053701 6997 raft_consensus.cc:348] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:11.053958 6997 raft_consensus.cc:729] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:11.054574 6997 consensus_queue.cc:260] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:11.055298 6997 ts_tablet_manager.cc:1398] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 20:11:11.055406 6547 raft_consensus.cc:1203] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.057230 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-2e97fe7ab97a4458b5e977af3d1356df received from {username='slave'} at 127.28.136.70:48699
I20240501 20:11:11.057380 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-2e97fe7ab97a4458b5e977af3d1356df on tablet 2e97fe7ab97a4458b5e977af3d1356df with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:11.066604 7008 tablet_copy_client.cc:523] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.070179 7008 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:11.073104 7028 tablet_copy_client.cc:523] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.076352 7028 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:11.082553 7234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:11.083591 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:11.083899 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:11.100353 7234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:11.114648 6787 raft_consensus.cc:1050] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:11.115022 7234 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:11.115168 7234 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:11.115305 7234 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:11.115393 7234 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:11.115761 6787 consensus_queue.cc:237] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:11.127571 6282 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.128269 4579 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.128937 4447 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.130439 6927 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.132334 6696 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.132930 6696 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.141290 6927 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.146804 7028 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.147409 7028 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:11.147774 7028 ts_tablet_manager.cc:1367] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.072s user 0.048s sys 0.010s
I20240501 20:11:11.148806 7028 raft_consensus.cc:348] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:11.148629 4579 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.149235 7028 raft_consensus.cc:729] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:11.149506 7028 consensus_queue.cc:260] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:11.150141 7028 ts_tablet_manager.cc:1398] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:11.150938 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-79cb3f8b3fe94497bb54d3d8b7f18754 received from {username='slave'} at 127.28.136.71:52891
I20240501 20:11:11.151063 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session 9592068f16ca406eb8a347a65d365782-79cb3f8b3fe94497bb54d3d8b7f18754 on tablet 79cb3f8b3fe94497bb54d3d8b7f18754 with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:11.151101 6282 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.155992 4447 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.156800 4030 catalog_manager.cc:5484] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.175765 4318 consensus_queue.cc:237] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:11.177939 4579 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.178279 6282 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.178814 6787 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.179450 7008 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.180250 6787 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.180788 7008 tablet_bootstrap.cc:492] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:11.181079 7008 ts_tablet_manager.cc:1367] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.111s user 0.044s sys 0.007s
I20240501 20:11:11.181404 6787 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.182080 7008 raft_consensus.cc:348] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:11.182343 7008 raft_consensus.cc:729] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:11.182543 7008 consensus_queue.cc:260] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:11.184022 7008 ts_tablet_manager.cc:1398] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:11.184233 6282 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.184656 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-bbd253f3e7824a81b6fdaf2c8880aaa9 received from {username='slave'} at 127.28.136.69:42579
I20240501 20:11:11.184803 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-bbd253f3e7824a81b6fdaf2c8880aaa9 on tablet bbd253f3e7824a81b6fdaf2c8880aaa9 with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:11.185668 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ff47913ab3b4771abbd9c912bbf34ad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:11.186270 4579 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.192307 4032 catalog_manager.cc:5484] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:11.198079 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff47913ab3b4771abbd9c912bbf34ad with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:11.200430 7376 tablet_replica.cc:330] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:11.200959 7376 raft_consensus.cc:2229] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:11.201432 7376 raft_consensus.cc:2258] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:11.202853 6282 raft_consensus.cc:1203] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.203313 7376 ts_tablet_manager.cc:1875] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:11.211439 7376 ts_tablet_manager.cc:1888] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:11.211666 7376 log.cc:1196] T 9ff47913ab3b4771abbd9c912bbf34ad P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/9ff47913ab3b4771abbd9c912bbf34ad
I20240501 20:11:11.214325 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 9ff47913ab3b4771abbd9c912bbf34ad (table rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]) successfully deleted
I20240501 20:11:11.217190 7072 tablet_copy_client.cc:523] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.220072 7072 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:11.222205 6148 raft_consensus.cc:1203] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.226687 7059 tablet_copy_client.cc:523] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.229513 7059 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:11.280665 6547 raft_consensus.cc:1203] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.283877 7090 tablet_copy_client.cc:523] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.290987 7090 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:11.304360 7059 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.305186 7059 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:11.305534 7059 ts_tablet_manager.cc:1367] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.076s user 0.059s sys 0.008s
I20240501 20:11:11.306453 7059 raft_consensus.cc:348] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:11.306751 7059 raft_consensus.cc:729] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:11.307018 7059 consensus_queue.cc:260] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:11.311288 7059 ts_tablet_manager.cc:1398] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:11:11.314131 7072 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.315487 7072 tablet_bootstrap.cc:492] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:11.321102 7072 ts_tablet_manager.cc:1367] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.101s user 0.044s sys 0.013s
I20240501 20:11:11.322094 7072 raft_consensus.cc:348] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:11.312637 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-60871b6059c249bfaac713e2f7325811 received from {username='slave'} at 127.28.136.71:50981
I20240501 20:11:11.323937 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session 9592068f16ca406eb8a347a65d365782-60871b6059c249bfaac713e2f7325811 on tablet 60871b6059c249bfaac713e2f7325811 with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:11.325223 7072 raft_consensus.cc:729] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:11.325548 7072 consensus_queue.cc:260] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:11.328063 7072 ts_tablet_manager.cc:1398] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 20:11:11.332057 7100 tablet_copy_client.cc:523] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.335366 7100 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:11.343894 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-28b9b57213a64172afdc2ca890569fc5 received from {username='slave'} at 127.28.136.72:50629
I20240501 20:11:11.344039 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-28b9b57213a64172afdc2ca890569fc5 on tablet 28b9b57213a64172afdc2ca890569fc5 with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:11.354955 7354 raft_consensus.cc:1050] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:11.355890 7354 consensus_queue.cc:237] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:11.358589 4185 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.358757 4579 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.359455 6787 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.359825 6148 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.360451 6696 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.364970 6787 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.372951 6416 raft_consensus.cc:1203] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.375079 6787 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.380501 4580 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.383369 6148 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.388821 4032 catalog_manager.cc:5484] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.392259 4185 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.402972 4318 consensus_queue.cc:237] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:11.406400 4185 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.407006 6787 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.410478 6148 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.412068 6787 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.413108 7354 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.418726 6148 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.421080 4185 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.425686 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0822a3fc82134d018bae96fecc62fdd3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:11.427984 4032 catalog_manager.cc:5484] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:11.436353 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 0822a3fc82134d018bae96fecc62fdd3 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:11.453859 7117 tablet_copy_client.cc:523] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.454291 7093 tablet_copy_client.cc:523] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.457418 7090 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.457477 7117 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:11.457834 7093 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:11.457841 7090 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:11.458106 7090 ts_tablet_manager.cc:1367] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.167s user 0.041s sys 0.008s
I20240501 20:11:11.458520 7441 tablet_replica.cc:330] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:11.459026 7441 raft_consensus.cc:2229] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:11.458896 7090 raft_consensus.cc:348] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:11.459154 7090 raft_consensus.cc:729] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:11.459445 7441 raft_consensus.cc:2258] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:11.459386 7090 consensus_queue.cc:260] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:11.461789 7441 ts_tablet_manager.cc:1875] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:11.463286 7090 ts_tablet_manager.cc:1398] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:11:11.464174 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-d09b0474ecbf4d458d7a520c9e9f329b received from {username='slave'} at 127.28.136.69:60381
I20240501 20:11:11.464324 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-d09b0474ecbf4d458d7a520c9e9f329b on tablet d09b0474ecbf4d458d7a520c9e9f329b with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:11.469559 7100 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.470049 7100 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:11.470817 7100 ts_tablet_manager.cc:1367] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.136s user 0.045s sys 0.003s
I20240501 20:11:11.471064 6202 raft_consensus.cc:1050] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:11.471783 7100 raft_consensus.cc:348] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:11.472131 7100 raft_consensus.cc:729] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:11.472143 6202 consensus_queue.cc:237] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:11.472393 7100 consensus_queue.cc:260] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:11.473690 7100 ts_tablet_manager.cc:1398] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:11.474402 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-96005b2bfa9847bb992640ad40a68b36 received from {username='slave'} at 127.28.136.69:60381
I20240501 20:11:11.474555 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-96005b2bfa9847bb992640ad40a68b36 on tablet 96005b2bfa9847bb992640ad40a68b36 with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:11.484975 7441 ts_tablet_manager.cc:1888] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:11.485265 7441 log.cc:1196] T 0822a3fc82134d018bae96fecc62fdd3 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/0822a3fc82134d018bae96fecc62fdd3
I20240501 20:11:11.486449 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 0822a3fc82134d018bae96fecc62fdd3 (table rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]) successfully deleted
I20240501 20:11:11.493235 4318 raft_consensus.cc:1261] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.494367 6547 raft_consensus.cc:1261] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.495112 6835 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.495759 6835 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.498694 4447 raft_consensus.cc:1261] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.501981 6835 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.508325 6835 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.512816 4318 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.518779 4026 catalog_manager.cc:5484] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:11.525034 6546 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.531188 6416 raft_consensus.cc:1203] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.548863 7117 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.549353 7117 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:11.549664 7117 ts_tablet_manager.cc:1367] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.092s user 0.050s sys 0.003s
I20240501 20:11:11.550544 7117 raft_consensus.cc:348] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:11.550866 7117 raft_consensus.cc:729] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:11.551090 7117 consensus_queue.cc:260] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:11.552626 7117 ts_tablet_manager.cc:1398] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:11.553514 4600 tablet_copy_service.cc:342] P 75b5acf03b9f4f6e98a67ae046587e79: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-d48a2b58ce2d417e9abf63c292419d14 received from {username='slave'} at 127.28.136.71:44331
I20240501 20:11:11.553661 4600 tablet_copy_service.cc:433] P 75b5acf03b9f4f6e98a67ae046587e79: ending tablet copy session 9592068f16ca406eb8a347a65d365782-d48a2b58ce2d417e9abf63c292419d14 on tablet d48a2b58ce2d417e9abf63c292419d14 with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:11.569026 6148 raft_consensus.cc:1203] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.573015 6148 raft_consensus.cc:1203] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.574020 7093 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.574501 7093 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:11.574826 7093 ts_tablet_manager.cc:1367] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.117s user 0.043s sys 0.004s
I20240501 20:11:11.575131 4447 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.575627 7093 raft_consensus.cc:348] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:11.575878 7093 raft_consensus.cc:729] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:11.576072 7093 consensus_queue.cc:260] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:11.576606 7093 ts_tablet_manager.cc:1398] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:11.582517 4469 tablet_copy_service.cc:342] P 596d3e24651b4787916397f573c09f1d: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-c3b4a011c5944a1db9dbc496e87d5768 received from {username='slave'} at 127.28.136.70:58253
I20240501 20:11:11.582660 4469 tablet_copy_service.cc:433] P 596d3e24651b4787916397f573c09f1d: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-c3b4a011c5944a1db9dbc496e87d5768 on tablet c3b4a011c5944a1db9dbc496e87d5768 with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:11.597782 7483 raft_consensus.cc:1050] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:11.598640 4185 consensus_queue.cc:237] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:11.599150 7483 consensus_queue.cc:237] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:11.600670 6547 raft_consensus.cc:1261] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.600915 4447 raft_consensus.cc:1261] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.601912 6600 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.603017 6927 raft_consensus.cc:1050] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:11.603838 6927 consensus_queue.cc:237] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:11.605520 6600 consensus_queue.cc:1035] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.607244 7460 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.608327 6547 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.610081 6416 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.610540 4579 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.611106 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a452749c5c44b87bb495983853cea3a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:11.611449 7264 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.611886 6282 raft_consensus.cc:1203] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.612039 7264 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.611820 4447 raft_consensus.cc:2932] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.617728 4028 catalog_manager.cc:5484] T 0a452749c5c44b87bb495983853cea3a P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.621670 6547 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.621745 4580 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.622911 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a452749c5c44b87bb495983853cea3a with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:11.623562 6696 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.624228 6696 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.625491 4185 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.628301 6787 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.630681 4185 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.633040 7264 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.641119 6696 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.642082 7486 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:11.645817 4185 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:11.645886 6546 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.652597 4026 catalog_manager.cc:5484] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.656553 6416 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:11.657277 4028 catalog_manager.cc:5484] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:11.657254 4580 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.669529 4318 consensus_queue.cc:237] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:11.673240 4580 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.673278 6547 raft_consensus.cc:1261] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.673810 7354 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.674105 6787 consensus_queue.cc:1035] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.680274 7354 raft_consensus.cc:1050] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:11.681367 7354 consensus_queue.cc:237] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:11.686196 6927 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.689671 6546 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.691596 7498 tablet_replica.cc:330] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:11.692282 7498 raft_consensus.cc:2229] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:11.692894 7498 raft_consensus.cc:2258] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:11.694155 4577 raft_consensus.cc:2932] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.697753 7498 ts_tablet_manager.cc:1875] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:11.697852 4578 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:11.700301 4447 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.701540 6148 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.706246 7354 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.706856 7354 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.707775 4447 consensus_queue.cc:237] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:11.709789 4580 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.709785 6416 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.710335 7486 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.710476 7133 tablet_copy_client.cc:523] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.710901 7486 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.714056 6416 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:11.714533 7133 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:11.715094 4028 catalog_manager.cc:5484] T b344760b82cb4bc09379d990f9558121 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:11.719929 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet b344760b82cb4bc09379d990f9558121 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:11.720978 4028 catalog_manager.cc:5484] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:11.721711 4186 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.722914 6696 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.724705 4580 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:11.733438 7264 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:11.736129 7507 raft_consensus.cc:1050] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:11.736966 7498 ts_tablet_manager.cc:1888] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:11.737357 7498 log.cc:1196] T 0a452749c5c44b87bb495983853cea3a P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/0a452749c5c44b87bb495983853cea3a
I20240501 20:11:11.738358 4165 tablet_service.cc:1514] Processing DeleteTablet for tablet bb36776ab8b74586aead4ecad56cc815 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:11.737061 7507 consensus_queue.cc:237] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:11.740619 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb36776ab8b74586aead4ecad56cc815 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 20:11:11.754209 4253 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:36125, user_credentials={real_user=slave}} blocked reactor thread for 57156us
I20240501 20:11:11.754528 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b344760b82cb4bc09379d990f9558121 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:11.755105 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet 0a452749c5c44b87bb495983853cea3a (table rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]) successfully deleted
I20240501 20:11:11.757404 6547 raft_consensus.cc:1203] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.765072 6282 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.765756 7508 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.766820 7508 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.767371 6148 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.769943 7163 tablet_copy_client.cc:523] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.772135 4185 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.773460 7163 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:11.773344 4447 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.778035 4028 catalog_manager.cc:5484] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:11.780519 7514 tablet_replica.cc:330] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:11.781666 7514 raft_consensus.cc:2229] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:11.786794 7514 pending_rounds.cc:62] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Trying to abort 1 pending ops.
I20240501 20:11:11.787487 7514 pending_rounds.cc:68] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022978136487198720 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b344760b82cb4bc09379d990f9558121" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } } }
I20240501 20:11:11.788570 7514 raft_consensus.cc:2894] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:11:11.788772 7514 raft_consensus.cc:2866] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:11:11.790745 7514 raft_consensus.cc:2258] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:11.793052 7514 ts_tablet_manager.cc:1875] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:11.796080 7172 tablet_copy_client.cc:523] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.796183 4449 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.798817 7507 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.799891 7172 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:11.801240 4580 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.801961 7173 tablet_copy_client.cc:523] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:11.805902 7173 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:11.807348 7507 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.814286 6787 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.819244 7514 ts_tablet_manager.cc:1888] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:11.818928 6282 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.820269 7514 log.cc:1196] T b344760b82cb4bc09379d990f9558121 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/b344760b82cb4bc09379d990f9558121
I20240501 20:11:11.820256 4028 catalog_manager.cc:5484] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:11.821708 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet b344760b82cb4bc09379d990f9558121 (table rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]) successfully deleted
I20240501 20:11:11.822402 6416 raft_consensus.cc:1203] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:11.823629 7514 tablet_replica.cc:330] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:11.823855 7514 raft_consensus.cc:2229] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:11.825258 4449 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.824337 7514 raft_consensus.cc:2258] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:11.834283 7514 ts_tablet_manager.cc:1875] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:11.856235 4579 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.859647 4317 consensus_queue.cc:237] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:11.862056 6282 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.862802 6696 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.864768 4579 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:11.865291 7507 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:11.866184 6696 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.868319 4577 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.869915 6282 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.874188 4028 catalog_manager.cc:5484] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:11.876365 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e97fe7ab97a4458b5e977af3d1356df with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:11.878151 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e97fe7ab97a4458b5e977af3d1356df with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:11.886792 7514 ts_tablet_manager.cc:1888] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:11.887080 7514 log.cc:1196] T bb36776ab8b74586aead4ecad56cc815 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/bb36776ab8b74586aead4ecad56cc815
I20240501 20:11:11.888010 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet bb36776ab8b74586aead4ecad56cc815 (table rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]) successfully deleted
I20240501 20:11:11.897754 6600 raft_consensus.cc:1050] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:11.898684 6600 consensus_queue.cc:237] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:11.905100 7461 raft_consensus.cc:1050] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:11.905997 7461 consensus_queue.cc:237] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:11.924693 7133 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.929105 7133 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:11.929420 7547 tablet_replica.cc:330] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:11.929548 7133 ts_tablet_manager.cc:1367] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.215s user 0.047s sys 0.003s
I20240501 20:11:11.930398 7133 raft_consensus.cc:348] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:11.930655 7133 raft_consensus.cc:729] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:11.930861 7133 consensus_queue.cc:260] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:11.931310 7547 raft_consensus.cc:2229] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:11.931519 7133 ts_tablet_manager.cc:1398] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:11.931825 7547 raft_consensus.cc:2258] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:11.932318 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-3f7fb3f245414d05a8e5d837076195c2 received from {username='slave'} at 127.28.136.70:39057
I20240501 20:11:11.932341 7552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.73:46209 every 8 connection(s)
I20240501 20:11:11.932497 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-3f7fb3f245414d05a8e5d837076195c2 on tablet 3f7fb3f245414d05a8e5d837076195c2 with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:11.932617 7234 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.73:46209
I20240501 20:11:11.934365 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240501 20:11:11.935921 6547 raft_consensus.cc:1261] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.937037 7460 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.939911 7547 ts_tablet_manager.cc:1875] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:11.941744 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 7234
I20240501 20:11:11.941972 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240501 20:11:11.948357 6416 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.949208 7459 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.958668 7173 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.959122 7173 tablet_bootstrap.cc:492] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:11.959405 7173 ts_tablet_manager.cc:1367] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.154s user 0.030s sys 0.017s
I20240501 20:11:11.961102 4317 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.962011 7459 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.965180 7173 raft_consensus.cc:348] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:11.965530 7173 raft_consensus.cc:729] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:11.966885 7173 consensus_queue.cc:260] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:11.970441 7173 ts_tablet_manager.cc:1398] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:11:11.972097 4449 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.973034 7460 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.979154 7547 ts_tablet_manager.cc:1888] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:11.980319 4600 tablet_copy_service.cc:342] P 75b5acf03b9f4f6e98a67ae046587e79: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-bbf736b8dba9474da5fb7ca9d9348ac7 received from {username='slave'} at 127.28.136.72:48463
I20240501 20:11:11.980468 4600 tablet_copy_service.cc:433] P 75b5acf03b9f4f6e98a67ae046587e79: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-bbf736b8dba9474da5fb7ca9d9348ac7 on tablet bbf736b8dba9474da5fb7ca9d9348ac7 with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:11.981631 7172 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:11.983562 7172 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:11.983891 7172 ts_tablet_manager.cc:1367] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.184s user 0.050s sys 0.011s
I20240501 20:11:11.984723 7172 raft_consensus.cc:348] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:11.985015 7172 raft_consensus.cc:729] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:11.986375 7172 consensus_queue.cc:260] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:11.986948 7547 log.cc:1196] T 2e97fe7ab97a4458b5e977af3d1356df P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/2e97fe7ab97a4458b5e977af3d1356df
I20240501 20:11:11.987020 7172 ts_tablet_manager.cc:1398] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:11.986110 4447 raft_consensus.cc:1261] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.989665 7461 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.989830 4577 raft_consensus.cc:1261] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:11.991035 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-5cdac8c0b92d442ebd84fb43af7e32fb received from {username='slave'} at 127.28.136.69:42579
I20240501 20:11:11.991076 6600 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:11.991185 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-5cdac8c0b92d442ebd84fb43af7e32fb on tablet 5cdac8c0b92d442ebd84fb43af7e32fb with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:11.995926 7553 heartbeater.cc:347] Connected to a master server at 127.28.136.126:33949
I20240501 20:11:11.996299 7553 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:11.997396 7553 heartbeater.cc:511] Master 127.28.136.126:33949 requested a full tablet report, sending...
I20240501 20:11:11.997450 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 2e97fe7ab97a4458b5e977af3d1356df (table rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]) successfully deleted
I20240501 20:11:11.998963 4028 ts_manager.cc:194] Registered new tserver with Master: b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209)
I20240501 20:11:12.000079 4028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.73:54645
I20240501 20:11:12.006157 7459 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.008314 7458 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.008910 6547 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.012444 4317 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.013850 4578 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.014714 4028 catalog_manager.cc:5484] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.016680 4447 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.019309 6416 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.025358 4026 catalog_manager.cc:5484] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.025346 4449 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.028654 6282 raft_consensus.cc:1203] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:12.032356 7163 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:12.032852 7163 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:12.033159 7163 ts_tablet_manager.cc:1367] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.260s user 0.044s sys 0.008s
I20240501 20:11:12.033972 7163 raft_consensus.cc:348] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:12.034317 7163 raft_consensus.cc:729] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:12.034277 4185 consensus_queue.cc:237] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:12.034610 7163 consensus_queue.cc:260] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:12.035218 7163 ts_tablet_manager.cc:1398] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:12.035888 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-3d7b1a47b1fe4e4b9a60b18b0b99056b received from {username='slave'} at 127.28.136.71:50981
I20240501 20:11:12.036022 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session 9592068f16ca406eb8a347a65d365782-3d7b1a47b1fe4e4b9a60b18b0b99056b on tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:12.037912 6416 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.038904 4317 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.039755 6835 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.040077 6600 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.043193 4185 consensus_queue.cc:237] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:12.043917 6835 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.045250 6547 raft_consensus.cc:1261] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.046202 7458 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.047498 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60871b6059c249bfaac713e2f7325811 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.047641 6416 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.049778 4318 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.050519 6600 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.051187 4025 catalog_manager.cc:5484] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.053220 4449 raft_consensus.cc:1261] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.054493 7459 consensus_queue.cc:1035] T 1564f07420a346dbaa77ef6a2a1a1b42 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.055852 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1564f07420a346dbaa77ef6a2a1a1b42 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.056190 6547 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.056422 4449 raft_consensus.cc:2932] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.058158 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 60871b6059c249bfaac713e2f7325811 with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:12.058513 7547 tablet_replica.cc:330] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:12.058739 7547 raft_consensus.cc:2229] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.059008 7547 raft_consensus.cc:2258] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.059916 7547 ts_tablet_manager.cc:1875] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.061952 4032 catalog_manager.cc:5484] T 1564f07420a346dbaa77ef6a2a1a1b42 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.065765 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 1564f07420a346dbaa77ef6a2a1a1b42 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:12.068207 7547 ts_tablet_manager.cc:1888] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.068327 7547 log.cc:1196] T 60871b6059c249bfaac713e2f7325811 P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/60871b6059c249bfaac713e2f7325811
I20240501 20:11:12.069006 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 60871b6059c249bfaac713e2f7325811 (table rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]) successfully deleted
I20240501 20:11:12.071862 7563 tablet_replica.cc:330] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:12.072067 7563 raft_consensus.cc:2229] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.072292 7563 raft_consensus.cc:2258] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.073170 7563 ts_tablet_manager.cc:1875] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.077512 7563 ts_tablet_manager.cc:1888] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.077641 7563 log.cc:1196] T 1564f07420a346dbaa77ef6a2a1a1b42 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/1564f07420a346dbaa77ef6a2a1a1b42
I20240501 20:11:12.078346 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 1564f07420a346dbaa77ef6a2a1a1b42 (table rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]) successfully deleted
I20240501 20:11:12.091388 7264 raft_consensus.cc:1050] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:12.092092 7264 consensus_queue.cc:237] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:12.094579 4578 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.094578 6282 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.094576 4185 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.095161 7483 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.095494 7510 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.095835 7264 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.098523 7510 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.099856 4578 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.102243 6282 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.105228 4032 catalog_manager.cc:5484] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.107477 4185 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.112993 4449 consensus_queue.cc:237] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:12.114827 6282 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.114900 4577 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.115404 7486 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.115679 7510 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.118005 7486 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.118672 4578 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.118854 6282 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.121577 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3b4a011c5944a1db9dbc496e87d5768 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.123854 4030 catalog_manager.cc:5484] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.125325 7459 raft_consensus.cc:1050] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:12.126093 7459 consensus_queue.cc:237] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.127905 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b4a011c5944a1db9dbc496e87d5768 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:12.128290 7514 tablet_replica.cc:330] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:12.128535 7514 raft_consensus.cc:2229] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.128865 7514 raft_consensus.cc:2258] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.129829 7514 ts_tablet_manager.cc:1875] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.130416 6148 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.130409 4318 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.131004 7459 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.131342 6835 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.132962 4449 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.133646 6835 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.136281 7514 ts_tablet_manager.cc:1888] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.136508 7514 log.cc:1196] T c3b4a011c5944a1db9dbc496e87d5768 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/c3b4a011c5944a1db9dbc496e87d5768
I20240501 20:11:12.137393 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet c3b4a011c5944a1db9dbc496e87d5768 (table rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]) successfully deleted
I20240501 20:11:12.144230 6600 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.145637 4318 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.147933 4449 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.147923 6148 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.149226 4030 catalog_manager.cc:5484] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.154536 4185 consensus_queue.cc:237] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.156869 6148 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.157349 7461 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.157588 7507 raft_consensus.cc:1050] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:12.158350 4449 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.158399 7507 consensus_queue.cc:237] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:12.160511 6416 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.160689 4449 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.160755 4578 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.161437 7508 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.161864 6835 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.162004 7508 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.161770 7461 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.162408 7508 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.163498 4449 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.165426 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96005b2bfa9847bb992640ad40a68b36 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.168798 6148 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.168951 4026 catalog_manager.cc:5484] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.174335 7459 raft_consensus.cc:1050] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:12.175295 7459 consensus_queue.cc:237] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.177850 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet 96005b2bfa9847bb992640ad40a68b36 with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:12.178208 7498 tablet_replica.cc:330] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:12.178427 7498 raft_consensus.cc:2229] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.178757 7498 raft_consensus.cc:2258] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.179633 6787 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.181463 4578 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.182560 7498 ts_tablet_manager.cc:1875] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.184134 4447 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.186921 4026 catalog_manager.cc:5484] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.187669 6416 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.192459 7498 ts_tablet_manager.cc:1888] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.192592 7498 log.cc:1196] T 96005b2bfa9847bb992640ad40a68b36 P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/96005b2bfa9847bb992640ad40a68b36
I20240501 20:11:12.194058 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet 96005b2bfa9847bb992640ad40a68b36 (table rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]) successfully deleted
I20240501 20:11:12.198452 6147 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.198961 4447 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.199065 7458 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.199471 7461 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.200024 4317 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.201141 7461 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.202765 4317 consensus_queue.cc:237] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:12.204497 4447 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.204591 6416 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.204617 6835 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.205629 6147 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.207718 6696 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.208083 6787 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.209563 4447 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.210269 4317 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.212234 6787 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.212285 4032 catalog_manager.cc:5484] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.212491 4449 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.216996 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.217031 6416 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.221305 4025 catalog_manager.cc:5484] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.222918 4185 consensus_queue.cc:237] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.226847 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:12.227386 7563 tablet_replica.cc:330] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:12.227610 7563 raft_consensus.cc:2229] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.227859 7563 raft_consensus.cc:2258] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.228943 7563 ts_tablet_manager.cc:1875] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.229072 4449 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.229645 7458 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.232656 7458 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.235502 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d09b0474ecbf4d458d7a520c9e9f329b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.236773 4447 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.237263 6148 raft_consensus.cc:1203] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:12.237423 4028 catalog_manager.cc:5484] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.241562 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet d09b0474ecbf4d458d7a520c9e9f329b with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:12.241962 7498 tablet_replica.cc:330] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:12.242194 7498 raft_consensus.cc:2229] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.242177 6147 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.242447 7498 raft_consensus.cc:2258] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.242488 7563 ts_tablet_manager.cc:1888] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.242619 7563 log.cc:1196] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/79cb3f8b3fe94497bb54d3d8b7f18754
I20240501 20:11:12.242695 6835 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.243333 7498 ts_tablet_manager.cc:1875] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.243332 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 79cb3f8b3fe94497bb54d3d8b7f18754 (table rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]) successfully deleted
I20240501 20:11:12.248008 6147 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.250943 7498 ts_tablet_manager.cc:1888] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.251080 7498 log.cc:1196] T d09b0474ecbf4d458d7a520c9e9f329b P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/d09b0474ecbf4d458d7a520c9e9f329b
I20240501 20:11:12.252359 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet d09b0474ecbf4d458d7a520c9e9f329b (table rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]) successfully deleted
I20240501 20:11:12.260255 7586 raft_consensus.cc:1050] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:12.261389 7586 consensus_queue.cc:237] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:12.264194 4317 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.264405 6416 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.264405 4447 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.265013 7019 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.265336 7020 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.265962 7020 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.280120 7019 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.280992 6416 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.284724 4318 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.285565 4028 catalog_manager.cc:5484] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.289530 6547 raft_consensus.cc:1203] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:12.290184 4447 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.296842 4578 consensus_queue.cc:237] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:12.299234 6416 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.299401 4318 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.300055 7593 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.300412 7507 raft_consensus.cc:1050] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:12.300359 7020 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.301236 7507 consensus_queue.cc:237] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:12.305574 7593 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.306677 4447 raft_consensus.cc:1261] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.307272 7354 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.310757 6547 raft_consensus.cc:1261] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.311424 6696 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.312655 4578 raft_consensus.cc:1261] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.313931 6416 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.318143 6696 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.319126 4318 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.319568 4032 catalog_manager.cc:5484] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.322924 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d48a2b58ce2d417e9abf63c292419d14 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.323974 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet d48a2b58ce2d417e9abf63c292419d14 with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:12.324306 7547 tablet_replica.cc:330] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:12.324488 7547 raft_consensus.cc:2229] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.324827 7547 raft_consensus.cc:2258] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.325613 7547 ts_tablet_manager.cc:1875] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.331980 7547 ts_tablet_manager.cc:1888] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.332095 7547 log.cc:1196] T d48a2b58ce2d417e9abf63c292419d14 P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/d48a2b58ce2d417e9abf63c292419d14
I20240501 20:11:12.332755 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet d48a2b58ce2d417e9abf63c292419d14 (table rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]) successfully deleted
I20240501 20:11:12.333622 7508 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.335950 4578 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.337515 4447 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.339679 6547 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.339891 4025 catalog_manager.cc:5484] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.349750 4318 consensus_queue.cc:237] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:12.352903 4447 raft_consensus.cc:1261] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.353484 6696 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.353538 6547 raft_consensus.cc:1261] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.354072 6696 consensus_queue.cc:1035] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.356195 7354 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.357439 4447 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.360421 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28b9b57213a64172afdc2ca890569fc5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.361409 6547 raft_consensus.cc:2932] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.362052 4028 catalog_manager.cc:5484] T 28b9b57213a64172afdc2ca890569fc5 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.374274 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b9b57213a64172afdc2ca890569fc5 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:12.374632 7563 tablet_replica.cc:330] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:12.374868 7563 raft_consensus.cc:2229] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.375211 7563 raft_consensus.cc:2258] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.376461 7563 ts_tablet_manager.cc:1875] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.381314 7563 ts_tablet_manager.cc:1888] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.381409 7563 log.cc:1196] T 28b9b57213a64172afdc2ca890569fc5 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/28b9b57213a64172afdc2ca890569fc5
I20240501 20:11:12.382046 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 28b9b57213a64172afdc2ca890569fc5 (table rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]) successfully deleted
I20240501 20:11:12.383473 6416 raft_consensus.cc:1203] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:12.421075 4318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9"
dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50670
I20240501 20:11:12.421334 4318 raft_consensus.cc:595] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.497347 7459 raft_consensus.cc:1050] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:12.498260 7459 consensus_queue.cc:237] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:12.500800 4318 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.500870 4578 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.500891 6282 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.501384 7459 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.501668 6835 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.502010 7458 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.506228 6835 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.509502 4578 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.510838 4318 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.514475 4025 catalog_manager.cc:5484] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.515064 6282 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:12.566881 4185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2"
dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36288
I20240501 20:11:12.567240 4185 raft_consensus.cc:595] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:12.579422 4318 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:12.581947 4185 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.581991 4447 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:12.582150 4252 consensus_peers.cc:485] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eeb1eec290ce4dcbb2afedd57e3b81da. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:12.582572 7508 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.582890 6696 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.585741 7354 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:12.586288 4185 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:12.586545 4447 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:12.590837 4028 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.604406 4185 consensus_queue.cc:237] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:12.606680 4318 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:12.606765 4115 consensus_peers.cc:485] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 618ea2abb259425babf56375f41eb81c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:12.606838 4578 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.607287 7458 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.607625 6835 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.610527 7458 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:12.611214 4318 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:12.613927 4578 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:12.617494 4030 catalog_manager.cc:5484] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index -1 to 4, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.634079 4578 consensus_queue.cc:237] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:12.637182 4185 raft_consensus.cc:1261] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.637255 4447 raft_consensus.cc:1261] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.638007 7593 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.638411 7019 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:12.643297 4513 consensus_peers.cc:485] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13d3d6724e3242478fef21f83dc94452. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:12.643297 7019 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:12.644179 4447 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:12.644970 4185 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:12.648581 4025 catalog_manager.cc:5484] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.664091 4185 consensus_queue.cc:237] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:12.666590 4447 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.666580 4578 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:12.667024 4116 consensus_peers.cc:485] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d320136fd1d4eb895362144ef6a5322. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:12.667443 7458 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.667744 7459 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.670647 6835 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:12.671245 4447 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:12.671355 4578 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:12.677048 7354 raft_consensus.cc:979] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: : Instructing follower ee4daecd449e4fc981fd1fe2b0e492c2 to start an election
I20240501 20:11:12.677251 7354 raft_consensus.cc:1067] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Signalling peer ee4daecd449e4fc981fd1fe2b0e492c2 to start an election
I20240501 20:11:12.677275 4025 catalog_manager.cc:5484] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.678575 6147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9"
dest_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2"
from {username='slave'} at 127.28.136.66:59835
I20240501 20:11:12.678843 6147 raft_consensus.cc:482] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.679045 6147 raft_consensus.cc:3037] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.681895 6147 raft_consensus.cc:504] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.683532 6147 leader_election.cc:288] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [CANDIDATE]: Term 2 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:12.684020 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9" candidate_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:12.684291 4318 raft_consensus.cc:3032] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.684386 4318 raft_consensus.cc:729] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:11:12.684787 4318 consensus_queue.cc:260] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.685312 4318 raft_consensus.cc:3037] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.685587 4185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9" candidate_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:12.685856 4185 raft_consensus.cc:3037] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.687366 4318 raft_consensus.cc:2454] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4daecd449e4fc981fd1fe2b0e492c2 in term 2.
I20240501 20:11:12.689113 4185 raft_consensus.cc:2454] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4daecd449e4fc981fd1fe2b0e492c2 in term 2.
I20240501 20:11:12.689632 6081 leader_election.cc:302] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7, ee4daecd449e4fc981fd1fe2b0e492c2; no voters:
I20240501 20:11:12.690052 7246 raft_consensus.cc:2781] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.690598 7246 raft_consensus.cc:686] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 LEADER]: Becoming Leader. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Running, Role: LEADER
I20240501 20:11:12.691131 7246 consensus_queue.cc:237] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.696436 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd253f3e7824a81b6fdaf2c8880aaa9" candidate_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:11:12.696199 4028 catalog_manager.cc:5484] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: term changed from 1 to 2, leader changed from 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) to ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69). New cstate: current_term: 2 leader_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.696658 4447 raft_consensus.cc:3037] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.698839 4447 raft_consensus.cc:2454] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4daecd449e4fc981fd1fe2b0e492c2 in term 2.
I20240501 20:11:12.716853 4318 consensus_queue.cc:237] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:12.719076 4185 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.719058 4447 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:12.719218 4252 consensus_peers.cc:485] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c11109bc41e34ab58fe3c9f568adf85e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:12.719589 7508 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.719916 7354 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.722811 7508 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:12.727468 4447 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:12.728402 7586 raft_consensus.cc:1050] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:12.729372 7586 consensus_queue.cc:237] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:12.730770 7354 raft_consensus.cc:1050] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:12.730746 4030 catalog_manager.cc:5484] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.731570 7354 consensus_queue.cc:237] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.734012 4185 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:12.741045 4578 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.741665 6147 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.741786 6696 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.742242 7507 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.743402 4447 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.744002 7354 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.766460 7507 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.768039 6547 raft_consensus.cc:1261] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.768335 4185 raft_consensus.cc:1261] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.768994 7019 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.769582 4318 raft_consensus.cc:1261] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.770231 7593 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.770797 7593 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.773656 6148 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.774503 4028 catalog_manager.cc:5484] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.778419 4578 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.784003 4317 consensus_queue.cc:237] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:12.786331 6147 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.787276 7507 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.787767 4577 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.789479 7354 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240501 20:11:12.790490 7019 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.792935 7354 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.794430 4449 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.795931 6546 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.799605 6147 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.802626 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cdac8c0b92d442ebd84fb43af7e32fb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.803495 4030 catalog_manager.cc:5484] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:12.805437 4028 catalog_manager.cc:5484] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.809306 4577 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:12.813504 4318 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.816087 4186 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.819339 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdac8c0b92d442ebd84fb43af7e32fb with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:12.819728 7547 tablet_replica.cc:330] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:12.819969 7547 raft_consensus.cc:2229] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.820222 7547 raft_consensus.cc:2258] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.821792 4186 consensus_queue.cc:237] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:12.822285 7547 ts_tablet_manager.cc:1875] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.824810 4577 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.824959 4318 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.825407 7460 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.825690 7459 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.828610 7547 ts_tablet_manager.cc:1888] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.828853 6835 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:12.829756 4577 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:12.829972 4318 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
W20240501 20:11:12.833199 4115 consensus_peers.cc:485] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca4d34ed427e44ed88379eb69feaeb8a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:12.837088 7547 log.cc:1196] T 5cdac8c0b92d442ebd84fb43af7e32fb P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/5cdac8c0b92d442ebd84fb43af7e32fb
I20240501 20:11:12.837896 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 5cdac8c0b92d442ebd84fb43af7e32fb (table rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]) successfully deleted
I20240501 20:11:12.845412 4030 catalog_manager.cc:5484] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:12.849889 4449 consensus_queue.cc:237] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:12.850334 4577 consensus_queue.cc:237] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:12.857496 4186 raft_consensus.cc:1261] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.858278 6546 raft_consensus.cc:1261] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.858256 7586 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.858919 7586 consensus_queue.cc:1035] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.860524 6835 raft_consensus.cc:1050] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:12.861433 6835 consensus_queue.cc:237] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:12.863026 7019 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.867861 4186 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.868968 6546 raft_consensus.cc:2932] T bbf736b8dba9474da5fb7ca9d9348ac7 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
W20240501 20:11:12.873230 4383 consensus_peers.cc:485] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6dc1dbd00224332941bc3301377eb87. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:12.874552 4318 raft_consensus.cc:1261] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.875192 7264 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.875334 4030 catalog_manager.cc:5484] T bbf736b8dba9474da5fb7ca9d9348ac7 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:12.879706 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet bbf736b8dba9474da5fb7ca9d9348ac7 with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:12.881990 6416 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.882596 4577 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.883414 7459 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:11:12.884008 7460 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.884961 4577 raft_consensus.cc:1261] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:12.885628 7264 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:12.886471 4447 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:12.886555 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bbf736b8dba9474da5fb7ca9d9348ac7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.887033 7458 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:12.906543 7458 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.912259 4577 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.915820 4025 catalog_manager.cc:5484] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.929152 4447 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.931958 7775 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:12.937701 7770 tablet_replica.cc:330] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:12.938134 7770 raft_consensus.cc:2229] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.938726 4578 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:12.939602 6416 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.941257 4186 consensus_queue.cc:237] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:12.944232 7770 raft_consensus.cc:2258] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.946203 4025 catalog_manager.cc:5484] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:12.947105 7770 ts_tablet_manager.cc:1875] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.950726 4449 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.951747 7458 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.954730 7458 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.958259 7770 ts_tablet_manager.cc:1888] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.960261 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:12.959690 4449 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.963034 7460 raft_consensus.cc:979] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: : Instructing follower f5195730e32a4abd8519baf9961d32f2 to start an election
I20240501 20:11:12.963183 7460 raft_consensus.cc:1067] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Signalling peer f5195730e32a4abd8519baf9961d32f2 to start an election
I20240501 20:11:12.963542 4030 catalog_manager.cc:5484] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:12.964213 6282 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2"
dest_uuid: "f5195730e32a4abd8519baf9961d32f2"
from {username='slave'} at 127.28.136.65:32967
I20240501 20:11:12.964499 6282 raft_consensus.cc:482] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:12.964710 6282 raft_consensus.cc:3037] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.967427 6282 raft_consensus.cc:504] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:12.969053 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:12.970937 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2" candidate_uuid: "f5195730e32a4abd8519baf9961d32f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:12.971164 4186 raft_consensus.cc:3032] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:12.971267 4186 raft_consensus.cc:729] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:11:12.971563 4186 consensus_queue.cc:260] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:12.972014 4186 raft_consensus.cc:3037] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.975014 4186 raft_consensus.cc:2454] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5195730e32a4abd8519baf9961d32f2 in term 2.
I20240501 20:11:12.975869 6282 leader_election.cc:288] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [CANDIDATE]: Term 2 election: Requested vote from peers 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:12.978881 7778 tablet_replica.cc:330] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:12.979130 7778 raft_consensus.cc:2229] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:12.979486 7778 raft_consensus.cc:2258] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:12.979647 6415 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:12.980073 7770 log.cc:1196] T bbf736b8dba9474da5fb7ca9d9348ac7 P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/bbf736b8dba9474da5fb7ca9d9348ac7
I20240501 20:11:12.981267 7778 ts_tablet_manager.cc:1875] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:12.982188 7458 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:12.983757 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet bbf736b8dba9474da5fb7ca9d9348ac7 (table rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]) successfully deleted
I20240501 20:11:12.983800 6415 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:12.953940 4318 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:12.987612 7778 ts_tablet_manager.cc:1888] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:12.987747 7778 log.cc:1196] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/3d7b1a47b1fe4e4b9a60b18b0b99056b
I20240501 20:11:12.987877 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2" candidate_uuid: "f5195730e32a4abd8519baf9961d32f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:12.988139 4317 raft_consensus.cc:3037] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:12.991448 4317 raft_consensus.cc:2454] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5195730e32a4abd8519baf9961d32f2 in term 2.
I20240501 20:11:12.991933 6217 leader_election.cc:302] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7, f5195730e32a4abd8519baf9961d32f2; no voters:
I20240501 20:11:12.992249 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b (table rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]) successfully deleted
I20240501 20:11:12.992448 7261 raft_consensus.cc:2781] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:12.997375 4578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7fb3f245414d05a8e5d837076195c2" candidate_uuid: "f5195730e32a4abd8519baf9961d32f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:11:12.997646 4578 raft_consensus.cc:3037] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:13.000182 4578 raft_consensus.cc:2454] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5195730e32a4abd8519baf9961d32f2 in term 2.
I20240501 20:11:13.000548 7261 raft_consensus.cc:686] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 2 LEADER]: Becoming Leader. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Running, Role: LEADER
I20240501 20:11:13.001207 7261 consensus_queue.cc:237] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:13.004666 7553 heartbeater.cc:503] Master 127.28.136.126:33949 was elected leader, sending a full tablet report...
I20240501 20:11:13.005754 4032 catalog_manager.cc:5484] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 reported cstate change: term changed from 1 to 2, leader changed from 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) to f5195730e32a4abd8519baf9961d32f2 (127.28.136.70). New cstate: current_term: 2 leader_uuid: "f5195730e32a4abd8519baf9961d32f2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:13.015918 4318 consensus_queue.cc:237] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:13.017411 7786 ts_tablet_manager.cc:902] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:13.024420 4578 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:13.025214 7507 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:11:13.025964 4251 consensus_peers.cc:485] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f240637b1e4ef8bc3414e11b38f469. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:13.029194 4447 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:13.029321 7790 ts_tablet_manager.cc:902] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:13.029860 7354 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:13.031659 7786 tablet_copy_client.cc:308] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36125
I20240501 20:11:13.032200 7507 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:13.034371 4578 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:13.039076 4447 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:13.039963 4032 catalog_manager.cc:5484] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:13.041121 7790 tablet_copy_client.cc:308] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:13.042574 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet eeb1eec290ce4dcbb2afedd57e3b81da from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:50629)
I20240501 20:11:13.042744 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet eeb1eec290ce4dcbb2afedd57e3b81da from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:50629: session id = 55fe2a4464a9402f927b2b60609b34dd-eeb1eec290ce4dcbb2afedd57e3b81da
I20240501 20:11:13.045459 4338 tablet_copy_source_session.cc:205] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.045729 4600 tablet_copy_service.cc:140] P 75b5acf03b9f4f6e98a67ae046587e79: Received BeginTabletCopySession request for tablet 13d3d6724e3242478fef21f83dc94452 from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:54795)
I20240501 20:11:13.045892 4600 tablet_copy_service.cc:161] P 75b5acf03b9f4f6e98a67ae046587e79: Beginning new tablet copy session on tablet 13d3d6724e3242478fef21f83dc94452 from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:54795: session id = ee4daecd449e4fc981fd1fe2b0e492c2-13d3d6724e3242478fef21f83dc94452
I20240501 20:11:13.047668 7790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb1eec290ce4dcbb2afedd57e3b81da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.048986 4600 tablet_copy_source_session.cc:205] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.050405 7786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d3d6724e3242478fef21f83dc94452. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.054080 7790 tablet_copy_client.cc:783] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.054407 7790 tablet_copy_client.cc:647] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.056234 7798 tablet_copy_client.cc:1002] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.058429 7786 tablet_copy_client.cc:783] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.058727 7786 tablet_copy_client.cc:647] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.060550 7794 tablet_copy_client.cc:1002] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.064924 4186 consensus_queue.cc:237] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:13.067512 4318 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:13.067767 4447 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:13.067787 4116 consensus_peers.cc:485] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 126f7b0e88774d51ac087e5a1f758c78. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:13.068291 7459 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:13.068609 6835 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:13.071533 6835 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:13.072175 4447 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:13.074054 4318 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:13.077383 4030 catalog_manager.cc:5484] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:13.121598 4185 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:13.122613 7246 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:13.163364 7849 ts_tablet_manager.cc:902] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:13.165387 7849 tablet_copy_client.cc:308] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:13.166095 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 618ea2abb259425babf56375f41eb81c from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:50981)
I20240501 20:11:13.166301 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 618ea2abb259425babf56375f41eb81c from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:50981: session id = 9592068f16ca406eb8a347a65d365782-618ea2abb259425babf56375f41eb81c
I20240501 20:11:13.168850 4206 tablet_copy_source_session.cc:205] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.170950 7849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ea2abb259425babf56375f41eb81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.176071 7849 tablet_copy_client.cc:783] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.176391 7849 tablet_copy_client.cc:647] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.178056 7858 tablet_copy_client.cc:1002] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.229072 4447 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:13.229713 7246 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:13.241779 4577 consensus_queue.cc:237] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:13.244410 4185 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:13.244498 4513 consensus_peers.cc:485] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae0fe7a1daab476391d80406766b3b11. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:13.245110 7586 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:13.245194 4447 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:13.245996 7586 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:13.248294 7019 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:13.249054 4185 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:13.253866 7902 ts_tablet_manager.cc:902] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:13.254462 4032 catalog_manager.cc:5484] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:13.258697 4447 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:13.260452 4318 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:13.261564 7898 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:13.291298 6147 consensus_queue.cc:237] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:13.293555 7902 tablet_copy_client.cc:308] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:13.293738 4447 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:13.293800 4185 raft_consensus.cc:1261] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:13.294464 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 7d320136fd1d4eb895362144ef6a5322 from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:39057)
I20240501 20:11:13.294446 7246 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:13.294617 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 7d320136fd1d4eb895362144ef6a5322 from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:39057: session id = f5195730e32a4abd8519baf9961d32f2-7d320136fd1d4eb895362144ef6a5322
I20240501 20:11:13.295006 7246 consensus_queue.cc:1035] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:13.297494 4206 tablet_copy_source_session.cc:205] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.297799 7898 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:13.298947 7902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d320136fd1d4eb895362144ef6a5322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.302924 4447 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:13.303144 4185 raft_consensus.cc:2932] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:13.305968 7902 tablet_copy_client.cc:783] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.306212 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bbd253f3e7824a81b6fdaf2c8880aaa9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:13.306680 7902 tablet_copy_client.cc:647] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.307848 4030 catalog_manager.cc:5484] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 5 to 7, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 2 leader_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:13.308357 7916 tablet_copy_client.cc:1002] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.312662 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd253f3e7824a81b6fdaf2c8880aaa9 with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:13.313130 7770 tablet_replica.cc:330] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:13.313318 7770 raft_consensus.cc:2229] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:13.313498 7770 raft_consensus.cc:2258] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:13.314764 7770 ts_tablet_manager.cc:1875] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:13.316026 7919 ts_tablet_manager.cc:902] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:13.325084 7770 ts_tablet_manager.cc:1888] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:11:13.325683 7770 log.cc:1196] T bbd253f3e7824a81b6fdaf2c8880aaa9 P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/bbd253f3e7824a81b6fdaf2c8880aaa9
I20240501 20:11:13.326675 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet bbd253f3e7824a81b6fdaf2c8880aaa9 (table rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]) successfully deleted
I20240501 20:11:13.329957 7919 tablet_copy_client.cc:308] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:13.330775 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet c11109bc41e34ab58fe3c9f568adf85e from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:50629)
I20240501 20:11:13.330964 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet c11109bc41e34ab58fe3c9f568adf85e from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:50629: session id = 55fe2a4464a9402f927b2b60609b34dd-c11109bc41e34ab58fe3c9f568adf85e
I20240501 20:11:13.333571 4338 tablet_copy_source_session.cc:205] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.334774 7919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11109bc41e34ab58fe3c9f568adf85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.340791 7919 tablet_copy_client.cc:783] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.341109 7919 tablet_copy_client.cc:647] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.342730 7926 tablet_copy_client.cc:1002] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.350852 7938 ts_tablet_manager.cc:902] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:13.352401 7938 tablet_copy_client.cc:308] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:13.353116 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet ca4d34ed427e44ed88379eb69feaeb8a from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:60381)
I20240501 20:11:13.353374 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet ca4d34ed427e44ed88379eb69feaeb8a from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:60381: session id = ee4daecd449e4fc981fd1fe2b0e492c2-ca4d34ed427e44ed88379eb69feaeb8a
I20240501 20:11:13.356616 4206 tablet_copy_source_session.cc:205] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.357936 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4d34ed427e44ed88379eb69feaeb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.362584 7938 tablet_copy_client.cc:783] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.362910 7938 tablet_copy_client.cc:647] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.364383 7943 tablet_copy_client.cc:1002] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.384951 4577 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:13.385639 7261 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:13.404978 4185 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:13.405628 7261 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:13.410229 4318 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:13.412019 7960 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:13.453114 7978 ts_tablet_manager.cc:902] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:13.480769 7978 tablet_copy_client.cc:308] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:13.481820 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 11f240637b1e4ef8bc3414e11b38f469 from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:42579)
I20240501 20:11:13.482005 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 11f240637b1e4ef8bc3414e11b38f469 from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:42579: session id = ee4daecd449e4fc981fd1fe2b0e492c2-11f240637b1e4ef8bc3414e11b38f469
I20240501 20:11:13.485276 7983 ts_tablet_manager.cc:902] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:13.485477 4338 tablet_copy_source_session.cc:205] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.487345 7978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f240637b1e4ef8bc3414e11b38f469. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.495324 7978 tablet_copy_client.cc:783] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.495678 7978 tablet_copy_client.cc:647] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.497625 7986 tablet_copy_client.cc:1002] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.498041 7983 tablet_copy_client.cc:308] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38865
I20240501 20:11:13.498860 4469 tablet_copy_service.cc:140] P 596d3e24651b4787916397f573c09f1d: Received BeginTabletCopySession request for tablet e6dc1dbd00224332941bc3301377eb87 from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:58253)
I20240501 20:11:13.499049 4469 tablet_copy_service.cc:161] P 596d3e24651b4787916397f573c09f1d: Beginning new tablet copy session on tablet e6dc1dbd00224332941bc3301377eb87 from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:58253: session id = f5195730e32a4abd8519baf9961d32f2-e6dc1dbd00224332941bc3301377eb87
I20240501 20:11:13.501655 4469 tablet_copy_source_session.cc:205] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.503357 7983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6dc1dbd00224332941bc3301377eb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.511377 7983 tablet_copy_client.cc:783] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.511654 7983 tablet_copy_client.cc:647] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.513505 7992 tablet_copy_client.cc:1002] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.513532 7999 ts_tablet_manager.cc:902] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:13.515166 7999 tablet_copy_client.cc:308] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:13.515990 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 126f7b0e88774d51ac087e5a1f758c78 from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:42689)
I20240501 20:11:13.516156 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 126f7b0e88774d51ac087e5a1f758c78 from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:42689: session id = 55fe2a4464a9402f927b2b60609b34dd-126f7b0e88774d51ac087e5a1f758c78
I20240501 20:11:13.518280 4206 tablet_copy_source_session.cc:205] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.519452 7999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126f7b0e88774d51ac087e5a1f758c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.524175 7999 tablet_copy_client.cc:783] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.524399 7999 tablet_copy_client.cc:647] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.525909 8003 tablet_copy_client.cc:1002] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.652115 8062 ts_tablet_manager.cc:902] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:13.653487 8062 tablet_copy_client.cc:308] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36125
I20240501 20:11:13.654114 4600 tablet_copy_service.cc:140] P 75b5acf03b9f4f6e98a67ae046587e79: Received BeginTabletCopySession request for tablet ae0fe7a1daab476391d80406766b3b11 from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:54795)
I20240501 20:11:13.654286 4600 tablet_copy_service.cc:161] P 75b5acf03b9f4f6e98a67ae046587e79: Beginning new tablet copy session on tablet ae0fe7a1daab476391d80406766b3b11 from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:54795: session id = ee4daecd449e4fc981fd1fe2b0e492c2-ae0fe7a1daab476391d80406766b3b11
I20240501 20:11:13.657006 4600 tablet_copy_source_session.cc:205] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:13.658283 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0fe7a1daab476391d80406766b3b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:13.663256 8062 tablet_copy_client.cc:783] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:13.663442 8062 tablet_copy_client.cc:647] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:13.664842 8066 tablet_copy_client.cc:1002] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:13.981015 6282 consensus_queue.cc:237] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:13.982663 4577 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:13.982717 4318 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:13.983158 8126 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:13.983500 8125 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:13.985491 8119 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:13.986140 4577 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:13.986181 4318 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:13.990756 4032 catalog_manager.cc:5484] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 2 leader_uuid: "f5195730e32a4abd8519baf9961d32f2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:13.992424 4019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f7fb3f245414d05a8e5d837076195c2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:13.996069 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f7fb3f245414d05a8e5d837076195c2 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:13.997807 8132 tablet_replica.cc:330] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:13.998013 8132 raft_consensus.cc:2229] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:13.998212 8132 raft_consensus.cc:2258] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:13.998975 8132 ts_tablet_manager.cc:1875] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:14.003381 8132 ts_tablet_manager.cc:1888] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:11:14.003479 8132 log.cc:1196] T 3f7fb3f245414d05a8e5d837076195c2 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/3f7fb3f245414d05a8e5d837076195c2
I20240501 20:11:14.004091 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet 3f7fb3f245414d05a8e5d837076195c2 (table rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]) successfully deleted
I20240501 20:11:14.556674 7790 tablet_copy_client.cc:523] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:14.560034 7790 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:14.560931 7786 tablet_copy_client.cc:523] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:14.563949 7786 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:14.602836 7790 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:14.603224 7790 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:14.603513 7790 ts_tablet_manager.cc:1367] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.044s user 0.038s sys 0.004s
I20240501 20:11:14.604173 7790 raft_consensus.cc:348] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:14.604388 7790 raft_consensus.cc:729] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:14.604625 7790 consensus_queue.cc:260] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:14.605736 7790 ts_tablet_manager.cc:1398] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:11:14.606530 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-eeb1eec290ce4dcbb2afedd57e3b81da received from {username='slave'} at 127.28.136.72:50629
I20240501 20:11:14.606685 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-eeb1eec290ce4dcbb2afedd57e3b81da on tablet eeb1eec290ce4dcbb2afedd57e3b81da with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:14.609660 7786 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:14.610042 7786 tablet_bootstrap.cc:492] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:14.610358 7786 ts_tablet_manager.cc:1367] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.046s user 0.033s sys 0.010s
I20240501 20:11:14.611191 7786 raft_consensus.cc:348] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:14.611479 7786 raft_consensus.cc:729] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:14.611688 7786 consensus_queue.cc:260] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:14.612282 7786 ts_tablet_manager.cc:1398] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:14.613088 4600 tablet_copy_service.cc:342] P 75b5acf03b9f4f6e98a67ae046587e79: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-13d3d6724e3242478fef21f83dc94452 received from {username='slave'} at 127.28.136.69:54795
I20240501 20:11:14.613224 4600 tablet_copy_service.cc:433] P 75b5acf03b9f4f6e98a67ae046587e79: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-13d3d6724e3242478fef21f83dc94452 on tablet 13d3d6724e3242478fef21f83dc94452 with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:14.678522 7849 tablet_copy_client.cc:523] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:14.681752 7849 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:14.686868 6547 raft_consensus.cc:1203] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:14.713840 7849 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:14.714221 7849 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:14.714450 7849 ts_tablet_manager.cc:1367] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.006s
I20240501 20:11:14.715059 7849 raft_consensus.cc:348] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:14.715291 7849 raft_consensus.cc:729] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:14.715482 7849 consensus_queue.cc:260] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:14.716138 7849 ts_tablet_manager.cc:1398] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:11:14.717022 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-618ea2abb259425babf56375f41eb81c received from {username='slave'} at 127.28.136.71:50981
I20240501 20:11:14.717185 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session 9592068f16ca406eb8a347a65d365782-618ea2abb259425babf56375f41eb81c on tablet 618ea2abb259425babf56375f41eb81c with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:14.722249 6147 raft_consensus.cc:1203] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:14.808840 7902 tablet_copy_client.cc:523] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:14.811841 7902 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:14.841470 7902 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:14.841878 7902 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:14.842130 7902 ts_tablet_manager.cc:1367] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.003s
I20240501 20:11:14.842841 7902 raft_consensus.cc:348] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:14.843114 7902 raft_consensus.cc:729] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:14.843153 7919 tablet_copy_client.cc:523] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:14.843320 7902 consensus_queue.cc:260] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:14.843916 7902 ts_tablet_manager.cc:1398] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:14.844666 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-7d320136fd1d4eb895362144ef6a5322 received from {username='slave'} at 127.28.136.70:39057
I20240501 20:11:14.844851 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-7d320136fd1d4eb895362144ef6a5322 on tablet 7d320136fd1d4eb895362144ef6a5322 with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:14.846503 7919 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:14.864809 7938 tablet_copy_client.cc:523] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:14.868491 7938 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:14.879480 7919 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:14.879859 7919 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:14.880137 7919 ts_tablet_manager.cc:1367] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.034s user 0.030s sys 0.001s
I20240501 20:11:14.880848 7919 raft_consensus.cc:348] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:14.881085 7919 raft_consensus.cc:729] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:14.881304 7919 consensus_queue.cc:260] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:14.881793 7919 ts_tablet_manager.cc:1398] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:11:14.882622 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-c11109bc41e34ab58fe3c9f568adf85e received from {username='slave'} at 127.28.136.72:50629
I20240501 20:11:14.882788 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-c11109bc41e34ab58fe3c9f568adf85e on tablet c11109bc41e34ab58fe3c9f568adf85e with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:14.910794 7938 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:14.911255 7938 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:14.911550 7938 ts_tablet_manager.cc:1367] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.043s user 0.039s sys 0.000s
I20240501 20:11:14.912343 7938 raft_consensus.cc:348] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:14.912602 7938 raft_consensus.cc:729] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:14.912813 7938 consensus_queue.cc:260] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:14.914130 7938 ts_tablet_manager.cc:1398] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:11:14.914866 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-ca4d34ed427e44ed88379eb69feaeb8a received from {username='slave'} at 127.28.136.69:60381
I20240501 20:11:14.915050 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-ca4d34ed427e44ed88379eb69feaeb8a on tablet ca4d34ed427e44ed88379eb69feaeb8a with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:14.998059 7978 tablet_copy_client.cc:523] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:15.001564 7978 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:15.013875 7983 tablet_copy_client.cc:523] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:15.017460 7983 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:15.026626 7999 tablet_copy_client.cc:523] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:15.029575 7999 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:15.045642 7978 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:15.046087 7978 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:15.046408 7978 ts_tablet_manager.cc:1367] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.045s user 0.034s sys 0.008s
I20240501 20:11:15.047253 7978 raft_consensus.cc:348] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:15.047549 7978 raft_consensus.cc:729] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:15.047772 7978 consensus_queue.cc:260] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:15.048362 7978 ts_tablet_manager.cc:1398] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:15.049890 4338 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-11f240637b1e4ef8bc3414e11b38f469 received from {username='slave'} at 127.28.136.69:42579
I20240501 20:11:15.050057 4338 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-11f240637b1e4ef8bc3414e11b38f469 on tablet 11f240637b1e4ef8bc3414e11b38f469 with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:15.069093 7983 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:15.069485 7983 tablet_bootstrap.cc:492] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:15.069736 7983 ts_tablet_manager.cc:1367] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.052s user 0.039s sys 0.001s
I20240501 20:11:15.070434 7983 raft_consensus.cc:348] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:15.070657 7983 raft_consensus.cc:729] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:15.070863 7983 consensus_queue.cc:260] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:15.071341 7983 ts_tablet_manager.cc:1398] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:15.072124 4469 tablet_copy_service.cc:342] P 596d3e24651b4787916397f573c09f1d: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-e6dc1dbd00224332941bc3301377eb87 received from {username='slave'} at 127.28.136.70:58253
I20240501 20:11:15.072274 4469 tablet_copy_service.cc:433] P 596d3e24651b4787916397f573c09f1d: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-e6dc1dbd00224332941bc3301377eb87 on tablet e6dc1dbd00224332941bc3301377eb87 with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:15.074158 7999 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:15.074611 7999 tablet_bootstrap.cc:492] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:15.074923 7999 ts_tablet_manager.cc:1367] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.045s user 0.028s sys 0.009s
I20240501 20:11:15.075753 7999 raft_consensus.cc:348] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:15.076031 7999 raft_consensus.cc:729] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:15.076260 7999 consensus_queue.cc:260] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:15.076957 7999 ts_tablet_manager.cc:1398] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 20:11:15.077795 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-126f7b0e88774d51ac087e5a1f758c78 received from {username='slave'} at 127.28.136.72:42689
I20240501 20:11:15.077993 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-126f7b0e88774d51ac087e5a1f758c78 on tablet 126f7b0e88774d51ac087e5a1f758c78 with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:15.116561 8295 raft_consensus.cc:1050] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:15.117244 8295 consensus_queue.cc:237] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:15.119597 6147 raft_consensus.cc:1261] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.119704 4185 raft_consensus.cc:1261] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.120354 4449 raft_consensus.cc:1261] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.120441 8134 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.121031 8296 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.121711 8134 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.126363 8296 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.128895 6547 raft_consensus.cc:1203] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:15.129637 6546 raft_consensus.cc:1203] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:15.129375 4449 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.130311 6416 raft_consensus.cc:1203] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:15.139001 4030 catalog_manager.cc:5484] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:15.138922 6147 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.139438 4186 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.148651 4577 consensus_queue.cc:237] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:15.150727 6147 raft_consensus.cc:1261] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.150833 4449 raft_consensus.cc:1261] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.151322 8296 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.151679 8134 consensus_queue.cc:1035] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.152274 6282 raft_consensus.cc:1203] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:15.153939 8296 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.154469 6147 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.156337 4449 raft_consensus.cc:2932] T 13d3d6724e3242478fef21f83dc94452 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.158025 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 13d3d6724e3242478fef21f83dc94452 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:15.159312 4025 catalog_manager.cc:5484] T 13d3d6724e3242478fef21f83dc94452 P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.163164 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d3d6724e3242478fef21f83dc94452 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:15.163961 8364 tablet_replica.cc:330] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:15.164217 8364 raft_consensus.cc:2229] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:15.164477 8364 raft_consensus.cc:2258] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:15.165182 8062 tablet_copy_client.cc:523] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:15.167812 8062 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:15.168957 8364 ts_tablet_manager.cc:1875] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:15.174049 8364 ts_tablet_manager.cc:1888] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:15.174151 8364 log.cc:1196] T 13d3d6724e3242478fef21f83dc94452 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/13d3d6724e3242478fef21f83dc94452
I20240501 20:11:15.174860 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet 13d3d6724e3242478fef21f83dc94452 (table rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]) successfully deleted
I20240501 20:11:15.186594 8366 raft_consensus.cc:1050] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:15.187458 8366 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:15.190002 6546 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.190057 4447 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.190533 4186 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.190701 8060 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.191311 8060 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.193372 8060 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.206182 8062 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:15.206174 8060 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.206719 8062 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:15.207159 8062 ts_tablet_manager.cc:1367] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.039s user 0.037s sys 0.004s
I20240501 20:11:15.207585 4447 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.208029 8062 raft_consensus.cc:348] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:15.208364 8062 raft_consensus.cc:729] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:15.208616 8062 consensus_queue.cc:260] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:15.210083 8062 ts_tablet_manager.cc:1398] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:15.211701 4186 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.212456 4600 tablet_copy_service.cc:342] P 75b5acf03b9f4f6e98a67ae046587e79: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-ae0fe7a1daab476391d80406766b3b11 received from {username='slave'} at 127.28.136.69:54795
I20240501 20:11:15.212580 4600 tablet_copy_service.cc:433] P 75b5acf03b9f4f6e98a67ae046587e79: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-ae0fe7a1daab476391d80406766b3b11 on tablet ae0fe7a1daab476391d80406766b3b11 with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:15.213238 4032 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:15.213996 6546 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.220616 4318 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:15.222594 6546 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.222754 4186 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.223119 8369 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.223510 8366 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.225627 8367 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.226262 6546 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.226240 4186 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.228705 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eeb1eec290ce4dcbb2afedd57e3b81da with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:15.230924 4030 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:15.235993 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb1eec290ce4dcbb2afedd57e3b81da with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:15.236662 8376 tablet_replica.cc:330] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:15.236930 8376 raft_consensus.cc:2229] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:15.237296 8376 raft_consensus.cc:2258] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:15.238366 8376 ts_tablet_manager.cc:1875] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:15.243417 8376 ts_tablet_manager.cc:1888] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:15.243587 8376 log.cc:1196] T eeb1eec290ce4dcbb2afedd57e3b81da P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/eeb1eec290ce4dcbb2afedd57e3b81da
I20240501 20:11:15.244403 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet eeb1eec290ce4dcbb2afedd57e3b81da (table rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]) successfully deleted
I20240501 20:11:15.329242 6147 raft_consensus.cc:1203] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:15.358152 6282 raft_consensus.cc:1203] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:15.400558 6147 raft_consensus.cc:1203] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:15.418892 6147 raft_consensus.cc:1203] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:15.534088 8366 raft_consensus.cc:1050] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:15.534960 8366 consensus_queue.cc:237] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:15.535306 8353 raft_consensus.cc:1050] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:15.536302 8353 consensus_queue.cc:237] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:15.538164 4447 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.538205 4185 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.538182 6546 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.538780 8369 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.539129 8060 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.539922 8367 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.540469 4318 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.540601 6547 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.541956 7458 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.542555 7458 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.550827 4449 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.552206 8353 raft_consensus.cc:1050] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:15.552582 7458 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.553120 8353 consensus_queue.cc:237] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:15.554217 8369 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.555624 4185 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.557114 4449 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.562197 4318 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.561772 8467 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.566637 4030 catalog_manager.cc:5484] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.574353 4030 catalog_manager.cc:5484] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.586161 4318 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.586907 8477 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.588047 4577 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.589067 8477 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.592487 6547 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.594660 8475 raft_consensus.cc:1050] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:15.596719 8475 consensus_queue.cc:237] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:15.603718 4317 consensus_queue.cc:237] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:15.604825 6282 raft_consensus.cc:1261] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.605881 8097 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.608907 4185 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.610641 6547 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.610641 8367 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.611339 8477 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:15.614892 8367 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.615986 4317 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:15.618623 8060 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.620105 6547 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.620149 4185 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.622035 4032 catalog_manager.cc:5484] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:15.627501 4032 catalog_manager.cc:5484] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd reported cstate change: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.632072 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet c11109bc41e34ab58fe3c9f568adf85e with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:15.632576 8376 tablet_replica.cc:330] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:15.632835 8376 raft_consensus.cc:2229] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:15.633082 8376 raft_consensus.cc:2258] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:15.634986 8376 ts_tablet_manager.cc:1875] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:15.637688 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c11109bc41e34ab58fe3c9f568adf85e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:15.650172 8376 ts_tablet_manager.cc:1888] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:15.650560 8376 log.cc:1196] T c11109bc41e34ab58fe3c9f568adf85e P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/c11109bc41e34ab58fe3c9f568adf85e
I20240501 20:11:15.651551 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet c11109bc41e34ab58fe3c9f568adf85e (table rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]) successfully deleted
I20240501 20:11:15.660480 4317 raft_consensus.cc:1261] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.660578 4577 raft_consensus.cc:1261] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.661463 8097 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.662058 8097 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:11:15.686580 4115 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:38865, user_credentials={real_user=slave}} blocked reactor thread for 105941us
I20240501 20:11:15.687932 6416 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.690372 4577 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:15.690456 4447 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.693107 6547 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:15.704357 8484 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.706341 8475 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.709908 4578 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.710126 4317 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.715653 4030 catalog_manager.cc:5484] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.716621 6282 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.720623 6416 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:15.724859 4447 consensus_queue.cc:237] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:15.729674 6282 raft_consensus.cc:1261] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.729945 4578 raft_consensus.cc:1261] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.730268 8097 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.730664 8475 consensus_queue.cc:1035] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.734308 8097 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.735663 4578 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.738647 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6dc1dbd00224332941bc3301377eb87 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:15.740911 6282 raft_consensus.cc:2932] T e6dc1dbd00224332941bc3301377eb87 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.742045 4025 catalog_manager.cc:5484] T e6dc1dbd00224332941bc3301377eb87 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:15.747187 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet e6dc1dbd00224332941bc3301377eb87 with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:15.747999 8503 tablet_replica.cc:330] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:15.748224 8503 raft_consensus.cc:2229] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:15.748452 8503 raft_consensus.cc:2258] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:15.749254 8503 ts_tablet_manager.cc:1875] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:15.754757 8503 ts_tablet_manager.cc:1888] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:15.754886 8503 log.cc:1196] T e6dc1dbd00224332941bc3301377eb87 P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/e6dc1dbd00224332941bc3301377eb87
I20240501 20:11:15.755667 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet e6dc1dbd00224332941bc3301377eb87 (table rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]) successfully deleted
I20240501 20:11:15.759445 4185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78"
dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36712
I20240501 20:11:15.759675 4185 raft_consensus.cc:595] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:15.844614 4185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78"
dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36762
I20240501 20:11:15.844874 4185 raft_consensus.cc:595] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:15.864859 8484 raft_consensus.cc:1050] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:15.865727 8484 consensus_queue.cc:237] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:15.868289 4447 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.868391 6282 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.868381 4578 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.869233 8479 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.869621 8478 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.870611 8353 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.890704 8479 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.893745 4449 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.897569 4578 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.901515 4032 catalog_manager.cc:5484] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.903976 6282 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.907801 4185 consensus_queue.cc:237] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:15.910179 4449 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.913120 6282 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:15.913134 8484 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.913620 8479 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:15.915859 8479 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.917479 4449 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.919709 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d320136fd1d4eb895362144ef6a5322 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:15.920532 6282 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:15.922251 4025 catalog_manager.cc:5484] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 5 to 6, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:15.927495 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d320136fd1d4eb895362144ef6a5322 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:15.932719 8296 raft_consensus.cc:1050] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:15.933611 8296 consensus_queue.cc:237] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:15.942442 8579 tablet_replica.cc:330] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:15.942734 8579 raft_consensus.cc:2229] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:15.943152 8579 raft_consensus.cc:2258] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:15.945166 8579 ts_tablet_manager.cc:1875] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:15.948493 8353 raft_consensus.cc:1050] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:15.949280 8353 consensus_queue.cc:237] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:15.951994 6147 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.952054 4578 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.952137 4317 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.952199 8579 ts_tablet_manager.cc:1888] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:15.952323 8579 log.cc:1196] T 7d320136fd1d4eb895362144ef6a5322 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d320136fd1d4eb895362144ef6a5322
I20240501 20:11:15.952603 8467 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.953238 8467 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.954283 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 7d320136fd1d4eb895362144ef6a5322 (table rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]) successfully deleted
I20240501 20:11:15.954527 8467 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.958613 8484 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.960677 4317 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.964419 4578 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.970278 4030 catalog_manager.cc:5484] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.970788 4449 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.970790 4185 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.971057 6148 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:15.971652 8578 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.972218 8501 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:15.973093 8296 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:11:15.973872 6147 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.988374 8296 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.989058 4449 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.990437 4185 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.993764 6148 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:15.995769 4032 catalog_manager.cc:5484] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:16.000674 8367 raft_consensus.cc:1050] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:16.001616 8367 consensus_queue.cc:237] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:16.005679 6147 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:16.006177 8369 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:16.008598 4449 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:16.009097 8369 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:16.010140 4577 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:16.010387 4578 consensus_queue.cc:237] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:16.012312 4447 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:16.012634 8369 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:16.012979 8578 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.015278 6147 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:16.015974 8578 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.018157 4447 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.021453 8501 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.022881 6147 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.026109 4032 catalog_manager.cc:5484] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.030541 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0fe7a1daab476391d80406766b3b11 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:16.031594 8594 tablet_replica.cc:330] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:16.032548 8369 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.033162 8594 raft_consensus.cc:2229] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:16.033605 8594 raft_consensus.cc:2258] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:16.034113 4447 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.035903 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae0fe7a1daab476391d80406766b3b11 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:16.037969 8594 ts_tablet_manager.cc:1875] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:16.038439 4578 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.042270 4032 catalog_manager.cc:5484] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.044064 8594 ts_tablet_manager.cc:1888] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:16.044198 8594 log.cc:1196] T ae0fe7a1daab476391d80406766b3b11 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae0fe7a1daab476391d80406766b3b11
I20240501 20:11:16.044962 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet ae0fe7a1daab476391d80406766b3b11 (table rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]) successfully deleted
I20240501 20:11:16.044615 6147 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.050268 4317 consensus_queue.cc:237] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:16.052564 4578 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:16.052721 6147 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:16.053474 8367 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.053848 8366 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.056303 8369 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.057003 4578 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.057560 6147 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.060932 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11f240637b1e4ef8bc3414e11b38f469 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:16.062721 4030 catalog_manager.cc:5484] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 5 to 6, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:16.066324 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f240637b1e4ef8bc3414e11b38f469 with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:16.066735 8376 tablet_replica.cc:330] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:16.066952 8376 raft_consensus.cc:2229] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:16.067214 8376 raft_consensus.cc:2258] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:16.068112 8376 ts_tablet_manager.cc:1875] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:16.072943 8376 ts_tablet_manager.cc:1888] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:16.073084 8376 log.cc:1196] T 11f240637b1e4ef8bc3414e11b38f469 P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/11f240637b1e4ef8bc3414e11b38f469
I20240501 20:11:16.073820 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 11f240637b1e4ef8bc3414e11b38f469 (table rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]) successfully deleted
I20240501 20:11:16.117667 8353 raft_consensus.cc:979] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: : Instructing follower 596d3e24651b4787916397f573c09f1d to start an election
I20240501 20:11:16.117849 8353 raft_consensus.cc:1067] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Signalling peer 596d3e24651b4787916397f573c09f1d to start an election
I20240501 20:11:16.118568 4447 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78"
dest_uuid: "596d3e24651b4787916397f573c09f1d"
from {username='slave'} at 127.28.136.65:44775
I20240501 20:11:16.118824 4447 raft_consensus.cc:482] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:16.118950 4447 raft_consensus.cc:3037] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:16.120998 4447 raft_consensus.cc:504] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:16.122321 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:16.122298 4185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:16.122566 4317 raft_consensus.cc:3037] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:16.122567 4185 raft_consensus.cc:3032] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:16.122658 4185 raft_consensus.cc:729] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:11:16.123031 4447 leader_election.cc:288] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 2 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003)
I20240501 20:11:16.122960 4185 consensus_queue.cc:260] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:16.123387 4185 raft_consensus.cc:3037] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:16.125397 4317 raft_consensus.cc:2454] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 2.
I20240501 20:11:16.126068 4185 raft_consensus.cc:2454] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 2.
I20240501 20:11:16.126492 4382 leader_election.cc:302] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7; no voters:
I20240501 20:11:16.126785 8495 raft_consensus.cc:2781] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:16.126979 8495 raft_consensus.cc:686] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 2 LEADER]: Becoming Leader. State: Replica: 596d3e24651b4787916397f573c09f1d, State: Running, Role: LEADER
I20240501 20:11:16.127333 8495 consensus_queue.cc:237] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:16.130452 6547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126f7b0e88774d51ac087e5a1f758c78" candidate_uuid: "596d3e24651b4787916397f573c09f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "55fe2a4464a9402f927b2b60609b34dd"
I20240501 20:11:16.130720 6547 raft_consensus.cc:3037] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:16.130585 4032 catalog_manager.cc:5484] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d reported cstate change: term changed from 1 to 2, leader changed from 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) to 596d3e24651b4787916397f573c09f1d (127.28.136.67). New cstate: current_term: 2 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:16.133124 6547 raft_consensus.cc:2454] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 596d3e24651b4787916397f573c09f1d in term 2.
I20240501 20:11:16.137428 4185 consensus_queue.cc:237] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:16.139135 6416 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:16.139447 4578 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:16.139649 8353 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.140010 8467 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.142072 8479 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:16.142695 4578 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:16.142696 6416 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:16.146371 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 618ea2abb259425babf56375f41eb81c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:16.147989 4032 catalog_manager.cc:5484] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 5 to 6, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:16.151892 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet 618ea2abb259425babf56375f41eb81c with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:16.152294 8503 tablet_replica.cc:330] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:16.152500 8503 raft_consensus.cc:2229] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:16.152711 8503 raft_consensus.cc:2258] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:16.153568 8503 ts_tablet_manager.cc:1875] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:16.157868 8503 ts_tablet_manager.cc:1888] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:16.157979 8503 log.cc:1196] T 618ea2abb259425babf56375f41eb81c P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/618ea2abb259425babf56375f41eb81c
I20240501 20:11:16.158661 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet 618ea2abb259425babf56375f41eb81c (table rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]) successfully deleted
I20240501 20:11:16.203722 4186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a"
dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36806
I20240501 20:11:16.203933 4186 raft_consensus.cc:595] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:16.252393 4186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a"
dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36828
I20240501 20:11:16.252578 4186 raft_consensus.cc:595] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:16.398932 8479 raft_consensus.cc:979] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: : Instructing follower 75b5acf03b9f4f6e98a67ae046587e79 to start an election
I20240501 20:11:16.399124 8479 raft_consensus.cc:1067] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Signalling peer 75b5acf03b9f4f6e98a67ae046587e79 to start an election
I20240501 20:11:16.399863 4578 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a"
dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
from {username='slave'} at 127.28.136.65:56567
I20240501 20:11:16.400106 4578 raft_consensus.cc:482] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:16.400213 4578 raft_consensus.cc:3037] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:16.402295 4578 raft_consensus.cc:504] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:16.403129 4578 leader_election.cc:288] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 2 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155)
I20240501 20:11:16.403421 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:16.403540 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:16.403658 4186 raft_consensus.cc:3032] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:16.403759 4186 raft_consensus.cc:729] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:11:16.403764 4317 raft_consensus.cc:3037] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:16.403801 6147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4d34ed427e44ed88379eb69feaeb8a" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2"
I20240501 20:11:16.404099 6147 raft_consensus.cc:3037] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:16.404028 4186 consensus_queue.cc:260] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:16.404377 4186 raft_consensus.cc:3037] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:16.406242 4186 raft_consensus.cc:2454] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 2.
I20240501 20:11:16.406517 4317 raft_consensus.cc:2454] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 2.
I20240501 20:11:16.406867 6147 raft_consensus.cc:2454] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 2.
I20240501 20:11:16.406937 4515 leader_election.cc:302] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 63c1e379865d42e78ea6b6b1d908ece7, 75b5acf03b9f4f6e98a67ae046587e79; no voters:
I20240501 20:11:16.407235 8296 raft_consensus.cc:2781] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:16.407375 8296 raft_consensus.cc:686] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 LEADER]: Becoming Leader. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Running, Role: LEADER
I20240501 20:11:16.407671 8296 consensus_queue.cc:237] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:16.410270 4026 catalog_manager.cc:5484] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: term changed from 1 to 2, leader changed from 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) to 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68). New cstate: current_term: 2 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:16.605069 4317 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:16.605607 8495 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.681206 6547 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:16.681768 8495 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.688509 4186 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:16.689891 8495 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.706838 4447 consensus_queue.cc:237] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:16.710562 6546 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:16.711472 4318 raft_consensus.cc:1261] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:16.711928 8665 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:16.712321 8667 consensus_queue.cc:1035] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:16.714560 8665 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:16.715112 6546 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:16.716917 4318 raft_consensus.cc:2932] T 126f7b0e88774d51ac087e5a1f758c78 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:16.718230 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 126f7b0e88774d51ac087e5a1f758c78 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:16.719863 4026 catalog_manager.cc:5484] T 126f7b0e88774d51ac087e5a1f758c78 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 2 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:16.723937 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet 126f7b0e88774d51ac087e5a1f758c78 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:16.725178 8677 tablet_replica.cc:330] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:16.725386 8677 raft_consensus.cc:2229] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:16.725750 8677 raft_consensus.cc:2258] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:16.727174 8677 ts_tablet_manager.cc:1875] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:16.732856 8677 ts_tablet_manager.cc:1888] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:11:16.732976 8677 log.cc:1196] T 126f7b0e88774d51ac087e5a1f758c78 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/126f7b0e88774d51ac087e5a1f758c78
I20240501 20:11:16.733664 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet 126f7b0e88774d51ac087e5a1f758c78 (table rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]) successfully deleted
I20240501 20:11:16.884069 4186 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:16.884665 8296 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.904764 4318 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:16.905470 8296 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:16.916469 6147 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:11:16.918743 8501 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:17.314841 4578 consensus_queue.cc:237] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:17.316829 4318 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:17.316884 6147 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:11:17.317523 8701 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.317922 8296 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.320560 8296 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:17.321391 4318 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:17.324247 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca4d34ed427e44ed88379eb69feaeb8a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:17.325598 6147 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:17.326730 4025 catalog_manager.cc:5484] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 5 to 7, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 2 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:17.331583 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4d34ed427e44ed88379eb69feaeb8a with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:17.332183 8705 tablet_replica.cc:330] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:17.332393 8705 raft_consensus.cc:2229] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:17.332651 8705 raft_consensus.cc:2258] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:17.333527 8705 ts_tablet_manager.cc:1875] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:17.338574 8705 ts_tablet_manager.cc:1888] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:11:17.338688 8705 log.cc:1196] T ca4d34ed427e44ed88379eb69feaeb8a P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/ca4d34ed427e44ed88379eb69feaeb8a
I20240501 20:11:17.339366 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet ca4d34ed427e44ed88379eb69feaeb8a (table rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]) successfully deleted
I20240501 20:11:17.533188 4428 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:17.545166 6261 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:17.553740 4558 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:17.554101 4296 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:11:17.577842 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:17.583876 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:17.585744 6128 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:17.616876 4164 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:17.633626 7420 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:17.712842 4318 consensus_queue.cc:237] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:17.715348 6147 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.715786 4578 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.715935 8765 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.716380 8369 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.719805 6147 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:17.724324 4029 catalog_manager.cc:5484] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:17.720901 8369 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
W20240501 20:11:17.730700 4253 consensus_peers.cc:485] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cdac8c0b92d442ebd84fb43af7e32fb. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.730540 4578 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:17.744935 4318 consensus_queue.cc:237] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:17.747223 4578 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:17.747421 4251 consensus_peers.cc:485] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e97fe7ab97a4458b5e977af3d1356df. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.747650 6282 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.747802 8769 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.748338 8769 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.750496 8769 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:17.751226 4578 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:17.756044 6282 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:17.756814 4029 catalog_manager.cc:5484] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:17.774073 4578 consensus_queue.cc:237] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:17.776412 4449 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.776891 6147 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.777105 8296 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.777400 8501 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.780411 8501 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:17.781090 4449 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:17.788025 4030 catalog_manager.cc:5484] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
W20240501 20:11:17.794162 4514 consensus_peers.cc:485] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae0fe7a1daab476391d80406766b3b11. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.795459 6148 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:17.816756 4578 consensus_queue.cc:237] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
W20240501 20:11:17.820956 4514 consensus_peers.cc:485] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1e01586b0ba42869481577ceacc84e8. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.821233 4186 raft_consensus.cc:1261] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:17.821676 4318 raft_consensus.cc:1261] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:17.821730 8790 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:17.822176 8501 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:17.824501 8790 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:17.825186 4186 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:17.826005 4318 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:17.831159 4026 catalog_manager.cc:5484] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index -1 to 4, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:17.850687 4578 consensus_queue.cc:237] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:17.853471 4318 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.854440 8790 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:11:17.857065 4513 consensus_peers.cc:485] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Couldn't send request to peer ee4daecd449e4fc981fd1fe2b0e492c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d48a2b58ce2d417e9abf63c292419d14. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.857502 8296 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:17.858100 4318 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:17.859756 6416 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.860451 8501 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.863520 6416 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:17.865276 4029 catalog_manager.cc:5484] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 6 to 7, NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) added. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } }
I20240501 20:11:17.886418 4449 consensus_queue.cc:237] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:17.889272 6416 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.894384 8812 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.904608 4577 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.906569 8813 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.907461 8812 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:17.911785 6415 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:17.913657 4029 catalog_manager.cc:5484] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:17.916837 4384 consensus_peers.cc:485] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb36776ab8b74586aead4ecad56cc815. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.944068 4578 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:17.953130 4186 consensus_queue.cc:237] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:17.956158 6148 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.957018 8353 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:11:17.957685 4115 consensus_peers.cc:485] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96005b2bfa9847bb992640ad40a68b36. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.959475 4449 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.960139 8353 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.960811 8353 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:17.978662 6148 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:17.981485 4030 catalog_manager.cc:5484] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:17.984894 4449 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:17.995622 4186 consensus_queue.cc:237] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:17.997834 6148 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:17.998059 4116 consensus_peers.cc:485] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d09b0474ecbf4d458d7a520c9e9f329b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:17.998436 4449 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:17.998399 8825 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:17.998973 8825 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.001143 8827 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.001725 6148 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.002444 4449 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.006985 4026 catalog_manager.cc:5484] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:18.020202 4318 consensus_queue.cc:237] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:18.022801 4449 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:18.022907 4252 consensus_peers.cc:485] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79cb3f8b3fe94497bb54d3d8b7f18754. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.022964 6416 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.023350 8765 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.023689 8369 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.026314 8769 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:18.027007 4449 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:18.027060 6416 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:18.032315 4026 catalog_manager.cc:5484] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 6 to 7, NON_VOTER 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } } }
I20240501 20:11:18.046377 4318 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:18.048600 4186 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:18.048849 4253 consensus_peers.cc:485] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eeb1eec290ce4dcbb2afedd57e3b81da. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.048857 6546 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.049283 8769 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.050100 8769 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.052126 8369 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:18.052789 4185 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:18.053534 6546 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:18.059697 4026 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:18.072034 4318 consensus_queue.cc:237] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:18.074533 6148 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.074677 4578 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.075215 8765 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.075666 8769 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:11:18.076025 4251 consensus_peers.cc:485] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11f240637b1e4ef8bc3414e11b38f469. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.078400 8369 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:18.078929 6148 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:18.079267 4578 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:18.084125 4030 catalog_manager.cc:5484] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 6 to 7, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:18.098171 4318 consensus_queue.cc:237] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:18.100528 6148 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:18.101306 4252 consensus_peers.cc:485] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0822a3fc82134d018bae96fecc62fdd3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.101719 8868 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.102079 8869 ts_tablet_manager.cc:902] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:18.102069 4185 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.102752 8765 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.104526 8869 tablet_copy_client.cc:308] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:18.111265 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 5cdac8c0b92d442ebd84fb43af7e32fb from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:45225)
I20240501 20:11:18.111461 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 5cdac8c0b92d442ebd84fb43af7e32fb from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:45225: session id = b662c3da2b1b4f3693920ccb680b8d12-5cdac8c0b92d442ebd84fb43af7e32fb
I20240501 20:11:18.113638 4338 tablet_copy_source_session.cc:205] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.114302 8369 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.114976 6148 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.119180 4026 catalog_manager.cc:5484] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:18.122514 4185 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.131364 8869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdac8c0b92d442ebd84fb43af7e32fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.137806 4185 consensus_queue.cc:237] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:18.140928 4449 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.141743 8825 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.143400 6282 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.143970 8825 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.144260 8869 tablet_copy_client.cc:783] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.144635 8869 tablet_copy_client.cc:647] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.146759 8874 tablet_copy_client.cc:1002] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.146864 8353 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:18.147501 6282 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:18.151719 4029 catalog_manager.cc:5484] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:18.157487 4118 consensus_peers.cc:485] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d320136fd1d4eb895362144ef6a5322. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.160173 4449 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:18.172726 4578 consensus_queue.cc:237] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
W20240501 20:11:18.174986 4513 consensus_peers.cc:485] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Couldn't send request to peer 9592068f16ca406eb8a347a65d365782. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca4d34ed427e44ed88379eb69feaeb8a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.175009 6148 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:11:18.175141 4318 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:11:18.175588 8790 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:18.175922 8501 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:18.178150 8296 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:18.178824 6148 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:18.178793 4318 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:18.184528 4030 catalog_manager.cc:5484] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 7 to 8, NON_VOTER 9592068f16ca406eb8a347a65d365782 (127.28.136.71) added. New cstate: current_term: 2 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } }
I20240501 20:11:18.197541 6282 consensus_queue.cc:237] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:18.204751 4578 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:11:18.204788 4318 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:11:18.205385 8898 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:18.205878 8897 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:18.210371 8898 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:18.211313 4318 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:18.212662 4578 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
W20240501 20:11:18.214993 6217 consensus_peers.cc:485] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f7fb3f245414d05a8e5d837076195c2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.216470 4026 catalog_manager.cc:5484] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 reported cstate change: config changed from index 7 to 8, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 2 leader_uuid: "f5195730e32a4abd8519baf9961d32f2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:18.231573 8909 ts_tablet_manager.cc:902] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:18.232088 4318 consensus_queue.cc:237] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:18.237226 8902 ts_tablet_manager.cc:902] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
W20240501 20:11:18.242481 4252 consensus_peers.cc:485] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d -> Peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Couldn't send request to peer f5195730e32a4abd8519baf9961d32f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c11109bc41e34ab58fe3c9f568adf85e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:18.244187 6546 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.244423 4185 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:18.245010 8769 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.245417 8868 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:18.248447 8902 tablet_copy_client.cc:308] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36125
I20240501 20:11:18.249236 4600 tablet_copy_service.cc:140] P 75b5acf03b9f4f6e98a67ae046587e79: Received BeginTabletCopySession request for tablet d1e01586b0ba42869481577ceacc84e8 from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:48463)
I20240501 20:11:18.249423 4600 tablet_copy_service.cc:161] P 75b5acf03b9f4f6e98a67ae046587e79: Beginning new tablet copy session on tablet d1e01586b0ba42869481577ceacc84e8 from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:48463: session id = 55fe2a4464a9402f927b2b60609b34dd-d1e01586b0ba42869481577ceacc84e8
I20240501 20:11:18.250541 8868 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.251165 8909 tablet_copy_client.cc:308] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36125
I20240501 20:11:18.251832 4599 tablet_copy_service.cc:140] P 75b5acf03b9f4f6e98a67ae046587e79: Received BeginTabletCopySession request for tablet ae0fe7a1daab476391d80406766b3b11 from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:33091)
I20240501 20:11:18.252053 4599 tablet_copy_service.cc:161] P 75b5acf03b9f4f6e98a67ae046587e79: Beginning new tablet copy session on tablet ae0fe7a1daab476391d80406766b3b11 from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:33091: session id = f5195730e32a4abd8519baf9961d32f2-ae0fe7a1daab476391d80406766b3b11
I20240501 20:11:18.251783 4185 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.252492 4600 tablet_copy_source_session.cc:205] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.254483 8902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e01586b0ba42869481577ceacc84e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.254861 4599 tablet_copy_source_session.cc:205] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.254730 6546 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.258064 8909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae0fe7a1daab476391d80406766b3b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.261826 4026 catalog_manager.cc:5484] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd reported cstate change: config changed from index 6 to 7, NON_VOTER f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } } }
I20240501 20:11:18.267602 8902 tablet_copy_client.cc:783] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.267933 8902 tablet_copy_client.cc:647] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.268788 8909 tablet_copy_client.cc:783] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.269752 8919 tablet_copy_client.cc:1002] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.269810 8909 tablet_copy_client.cc:647] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.271368 8920 tablet_copy_client.cc:1002] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.275043 8926 ts_tablet_manager.cc:902] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:18.276786 8926 tablet_copy_client.cc:308] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38865
I20240501 20:11:18.283849 4469 tablet_copy_service.cc:140] P 596d3e24651b4787916397f573c09f1d: Received BeginTabletCopySession request for tablet bb36776ab8b74586aead4ecad56cc815 from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:40053)
I20240501 20:11:18.284013 4469 tablet_copy_service.cc:161] P 596d3e24651b4787916397f573c09f1d: Beginning new tablet copy session on tablet bb36776ab8b74586aead4ecad56cc815 from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:40053: session id = b662c3da2b1b4f3693920ccb680b8d12-bb36776ab8b74586aead4ecad56cc815
I20240501 20:11:18.286659 4469 tablet_copy_source_session.cc:205] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.288136 8926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb36776ab8b74586aead4ecad56cc815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.293493 8926 tablet_copy_client.cc:783] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.293747 8926 tablet_copy_client.cc:647] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.295498 8930 tablet_copy_client.cc:1002] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.302006 8946 ts_tablet_manager.cc:902] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:18.304898 8946 tablet_copy_client.cc:308] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:18.305616 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 2e97fe7ab97a4458b5e977af3d1356df from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:52891)
I20240501 20:11:18.305784 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 2e97fe7ab97a4458b5e977af3d1356df from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:52891: session id = 9592068f16ca406eb8a347a65d365782-2e97fe7ab97a4458b5e977af3d1356df
I20240501 20:11:18.308216 4338 tablet_copy_source_session.cc:205] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.309501 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e97fe7ab97a4458b5e977af3d1356df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.315348 8946 tablet_copy_client.cc:783] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.315572 8946 tablet_copy_client.cc:647] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.316974 8950 tablet_copy_client.cc:1002] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.335407 8968 ts_tablet_manager.cc:902] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: Initiating tablet copy from peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:18.337190 8968 tablet_copy_client.cc:308] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36125
I20240501 20:11:18.337965 4599 tablet_copy_service.cc:140] P 75b5acf03b9f4f6e98a67ae046587e79: Received BeginTabletCopySession request for tablet d48a2b58ce2d417e9abf63c292419d14 from peer ee4daecd449e4fc981fd1fe2b0e492c2 ({username='slave'} at 127.28.136.69:54795)
I20240501 20:11:18.338147 4599 tablet_copy_service.cc:161] P 75b5acf03b9f4f6e98a67ae046587e79: Beginning new tablet copy session on tablet d48a2b58ce2d417e9abf63c292419d14 from peer ee4daecd449e4fc981fd1fe2b0e492c2 at {username='slave'} at 127.28.136.69:54795: session id = ee4daecd449e4fc981fd1fe2b0e492c2-d48a2b58ce2d417e9abf63c292419d14
I20240501 20:11:18.340873 4599 tablet_copy_source_session.cc:205] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.342262 8968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48a2b58ce2d417e9abf63c292419d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.347369 8981 ts_tablet_manager.cc:902] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:18.347787 8968 tablet_copy_client.cc:783] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.348063 8968 tablet_copy_client.cc:647] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.349656 8975 tablet_copy_client.cc:1002] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.353715 8981 tablet_copy_client.cc:308] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:18.354355 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 96005b2bfa9847bb992640ad40a68b36 from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:50981)
I20240501 20:11:18.354544 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 96005b2bfa9847bb992640ad40a68b36 from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:50981: session id = 9592068f16ca406eb8a347a65d365782-96005b2bfa9847bb992640ad40a68b36
I20240501 20:11:18.357263 4206 tablet_copy_source_session.cc:205] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.358595 8981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96005b2bfa9847bb992640ad40a68b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.364779 8981 tablet_copy_client.cc:783] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.365006 8981 tablet_copy_client.cc:647] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.366379 8985 tablet_copy_client.cc:1002] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.457185 9021 ts_tablet_manager.cc:902] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:18.459364 9021 tablet_copy_client.cc:308] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:18.460206 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet d09b0474ecbf4d458d7a520c9e9f329b from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:39057)
I20240501 20:11:18.460414 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet d09b0474ecbf4d458d7a520c9e9f329b from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:39057: session id = f5195730e32a4abd8519baf9961d32f2-d09b0474ecbf4d458d7a520c9e9f329b
I20240501 20:11:18.462821 4206 tablet_copy_source_session.cc:205] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.464202 9021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09b0474ecbf4d458d7a520c9e9f329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.470538 9021 tablet_copy_client.cc:783] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.470783 9021 tablet_copy_client.cc:647] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.472327 9025 tablet_copy_client.cc:1002] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.530344 9026 ts_tablet_manager.cc:902] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:18.531711 9026 tablet_copy_client.cc:308] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:18.537004 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 7d320136fd1d4eb895362144ef6a5322 from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:53095)
I20240501 20:11:18.537185 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 7d320136fd1d4eb895362144ef6a5322 from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:53095: session id = b662c3da2b1b4f3693920ccb680b8d12-7d320136fd1d4eb895362144ef6a5322
I20240501 20:11:18.539319 4206 tablet_copy_source_session.cc:205] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.540632 9026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d320136fd1d4eb895362144ef6a5322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.545037 9026 tablet_copy_client.cc:783] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.545246 9026 tablet_copy_client.cc:647] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.546770 9030 tablet_copy_client.cc:1002] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.553452 9031 ts_tablet_manager.cc:902] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:18.555128 9031 tablet_copy_client.cc:308] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:18.555913 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 11f240637b1e4ef8bc3414e11b38f469 from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:52891)
I20240501 20:11:18.556093 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 11f240637b1e4ef8bc3414e11b38f469 from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:52891: session id = 9592068f16ca406eb8a347a65d365782-11f240637b1e4ef8bc3414e11b38f469
I20240501 20:11:18.558219 4338 tablet_copy_source_session.cc:205] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.559577 9031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11f240637b1e4ef8bc3414e11b38f469. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.564152 9031 tablet_copy_client.cc:783] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.564383 9031 tablet_copy_client.cc:647] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.566228 9036 ts_tablet_manager.cc:902] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:18.566282 9035 tablet_copy_client.cc:1002] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.567734 9036 tablet_copy_client.cc:308] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:18.568428 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 from peer 55fe2a4464a9402f927b2b60609b34dd ({username='slave'} at 127.28.136.72:50629)
I20240501 20:11:18.568609 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 79cb3f8b3fe94497bb54d3d8b7f18754 from peer 55fe2a4464a9402f927b2b60609b34dd at {username='slave'} at 127.28.136.72:50629: session id = 55fe2a4464a9402f927b2b60609b34dd-79cb3f8b3fe94497bb54d3d8b7f18754
I20240501 20:11:18.571209 4338 tablet_copy_source_session.cc:205] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.572237 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79cb3f8b3fe94497bb54d3d8b7f18754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.576543 9036 tablet_copy_client.cc:783] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.576792 9036 tablet_copy_client.cc:647] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.578297 9040 tablet_copy_client.cc:1002] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.624002 9041 ts_tablet_manager.cc:902] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:18.625413 9041 tablet_copy_client.cc:308] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:18.626138 4338 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 0822a3fc82134d018bae96fecc62fdd3 from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:48699)
I20240501 20:11:18.626344 4338 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 0822a3fc82134d018bae96fecc62fdd3 from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:48699: session id = f5195730e32a4abd8519baf9961d32f2-0822a3fc82134d018bae96fecc62fdd3
I20240501 20:11:18.626870 9046 ts_tablet_manager.cc:902] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:18.628319 9046 tablet_copy_client.cc:308] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:18.628598 4338 tablet_copy_source_session.cc:205] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.629032 4337 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet eeb1eec290ce4dcbb2afedd57e3b81da from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:45225)
I20240501 20:11:18.629179 4337 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet eeb1eec290ce4dcbb2afedd57e3b81da from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:45225: session id = b662c3da2b1b4f3693920ccb680b8d12-eeb1eec290ce4dcbb2afedd57e3b81da
I20240501 20:11:18.629957 9041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0822a3fc82134d018bae96fecc62fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.631808 4337 tablet_copy_source_session.cc:205] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.633183 9046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeb1eec290ce4dcbb2afedd57e3b81da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.635851 9041 tablet_copy_client.cc:783] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.636103 9041 tablet_copy_client.cc:647] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.637916 9045 tablet_copy_client.cc:1002] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.639964 9046 tablet_copy_client.cc:783] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.640208 9046 tablet_copy_client.cc:647] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.641841 9050 tablet_copy_client.cc:1002] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.761380 9122 ts_tablet_manager.cc:902] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983)
I20240501 20:11:18.764222 9126 ts_tablet_manager.cc:902] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: Initiating tablet copy from peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125)
I20240501 20:11:18.768108 9122 tablet_copy_client.cc:308] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:40983
I20240501 20:11:18.776186 6303 tablet_copy_service.cc:140] P f5195730e32a4abd8519baf9961d32f2: Received BeginTabletCopySession request for tablet 3f7fb3f245414d05a8e5d837076195c2 from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:33971)
I20240501 20:11:18.776432 6303 tablet_copy_service.cc:161] P f5195730e32a4abd8519baf9961d32f2: Beginning new tablet copy session on tablet 3f7fb3f245414d05a8e5d837076195c2 from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:33971: session id = b662c3da2b1b4f3693920ccb680b8d12-3f7fb3f245414d05a8e5d837076195c2
I20240501 20:11:18.781312 6303 tablet_copy_source_session.cc:205] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.797260 9131 ts_tablet_manager.cc:902] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:18.797719 9122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7fb3f245414d05a8e5d837076195c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.811444 9122 tablet_copy_client.cc:783] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.811749 9122 tablet_copy_client.cc:647] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.814343 9130 tablet_copy_client.cc:1002] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.823834 9126 tablet_copy_client.cc:308] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36125
I20240501 20:11:18.824718 4599 tablet_copy_service.cc:140] P 75b5acf03b9f4f6e98a67ae046587e79: Received BeginTabletCopySession request for tablet ca4d34ed427e44ed88379eb69feaeb8a from peer 9592068f16ca406eb8a347a65d365782 ({username='slave'} at 127.28.136.71:44331)
I20240501 20:11:18.824928 4599 tablet_copy_service.cc:161] P 75b5acf03b9f4f6e98a67ae046587e79: Beginning new tablet copy session on tablet ca4d34ed427e44ed88379eb69feaeb8a from peer 9592068f16ca406eb8a347a65d365782 at {username='slave'} at 127.28.136.71:44331: session id = 9592068f16ca406eb8a347a65d365782-ca4d34ed427e44ed88379eb69feaeb8a
I20240501 20:11:18.828066 4599 tablet_copy_source_session.cc:205] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.830080 9126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4d34ed427e44ed88379eb69feaeb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.833495 9131 tablet_copy_client.cc:308] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:18.834283 4337 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet c11109bc41e34ab58fe3c9f568adf85e from peer f5195730e32a4abd8519baf9961d32f2 ({username='slave'} at 127.28.136.70:48699)
I20240501 20:11:18.834460 4337 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet c11109bc41e34ab58fe3c9f568adf85e from peer f5195730e32a4abd8519baf9961d32f2 at {username='slave'} at 127.28.136.70:48699: session id = f5195730e32a4abd8519baf9961d32f2-c11109bc41e34ab58fe3c9f568adf85e
I20240501 20:11:18.836977 4337 tablet_copy_source_session.cc:205] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:18.838922 9131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11109bc41e34ab58fe3c9f568adf85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:18.839620 9126 tablet_copy_client.cc:783] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.839988 9126 tablet_copy_client.cc:647] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.842545 9142 tablet_copy_client.cc:1002] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:18.846302 9131 tablet_copy_client.cc:783] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:18.846565 9131 tablet_copy_client.cc:647] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:18.848366 9145 tablet_copy_client.cc:1002] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:19.647300 8869 tablet_copy_client.cc:523] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:19.651461 8869 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:19.661988 8869 log.cc:824] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:19.697378 8869 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:19.697846 8869 tablet_bootstrap.cc:492] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:19.698261 8869 ts_tablet_manager.cc:1367] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.047s user 0.036s sys 0.012s
I20240501 20:11:19.702620 8869 raft_consensus.cc:348] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:19.703032 8869 raft_consensus.cc:729] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:19.703495 8869 consensus_queue.cc:260] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:19.705950 8869 ts_tablet_manager.cc:1398] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240501 20:11:19.706881 4337 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-5cdac8c0b92d442ebd84fb43af7e32fb received from {username='slave'} at 127.28.136.73:45225
I20240501 20:11:19.707016 4337 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-5cdac8c0b92d442ebd84fb43af7e32fb on tablet 5cdac8c0b92d442ebd84fb43af7e32fb with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:19.770138 8902 tablet_copy_client.cc:523] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:19.771734 8909 tablet_copy_client.cc:523] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:19.773083 8902 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:19.775102 8909 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:19.795924 8926 tablet_copy_client.cc:523] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:19.798475 8926 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:19.817574 8946 tablet_copy_client.cc:523] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:19.821967 8909 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:19.822341 8946 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:19.822409 8909 tablet_bootstrap.cc:492] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:19.822683 8909 ts_tablet_manager.cc:1367] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.048s user 0.040s sys 0.007s
I20240501 20:11:19.823539 8909 raft_consensus.cc:348] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:19.823838 8909 raft_consensus.cc:729] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:19.824060 8909 consensus_queue.cc:260] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:19.824719 8909 ts_tablet_manager.cc:1398] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:11:19.825371 4599 tablet_copy_service.cc:342] P 75b5acf03b9f4f6e98a67ae046587e79: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-ae0fe7a1daab476391d80406766b3b11 received from {username='slave'} at 127.28.136.70:33091
I20240501 20:11:19.825536 4599 tablet_copy_service.cc:433] P 75b5acf03b9f4f6e98a67ae046587e79: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-ae0fe7a1daab476391d80406766b3b11 on tablet ae0fe7a1daab476391d80406766b3b11 with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:19.850106 8968 tablet_copy_client.cc:523] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:19.853973 8968 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap starting.
I20240501 20:11:19.866787 8981 tablet_copy_client.cc:523] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:19.870716 8981 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:19.874163 8926 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:19.876320 8926 tablet_bootstrap.cc:492] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:19.875173 8902 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:19.877099 8902 tablet_bootstrap.cc:492] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:19.877400 8902 ts_tablet_manager.cc:1367] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.104s user 0.045s sys 0.002s
I20240501 20:11:19.878602 8926 ts_tablet_manager.cc:1367] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.080s user 0.045s sys 0.011s
I20240501 20:11:19.880892 8902 raft_consensus.cc:348] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:19.881188 8902 raft_consensus.cc:729] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:19.882443 8902 consensus_queue.cc:260] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:19.880791 8926 raft_consensus.cc:348] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:19.882966 8926 raft_consensus.cc:729] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:19.887655 8902 ts_tablet_manager.cc:1398] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20240501 20:11:19.883198 8926 consensus_queue.cc:260] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:19.891505 8926 ts_tablet_manager.cc:1398] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 20:11:19.892323 4469 tablet_copy_service.cc:342] P 596d3e24651b4787916397f573c09f1d: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-bb36776ab8b74586aead4ecad56cc815 received from {username='slave'} at 127.28.136.73:40053
I20240501 20:11:19.892455 4469 tablet_copy_service.cc:433] P 596d3e24651b4787916397f573c09f1d: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-bb36776ab8b74586aead4ecad56cc815 on tablet bb36776ab8b74586aead4ecad56cc815 with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:19.902781 4599 tablet_copy_service.cc:342] P 75b5acf03b9f4f6e98a67ae046587e79: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-d1e01586b0ba42869481577ceacc84e8 received from {username='slave'} at 127.28.136.72:48463
I20240501 20:11:19.902947 4599 tablet_copy_service.cc:433] P 75b5acf03b9f4f6e98a67ae046587e79: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-d1e01586b0ba42869481577ceacc84e8 on tablet d1e01586b0ba42869481577ceacc84e8 with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:19.934626 8946 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:19.935143 8946 tablet_bootstrap.cc:492] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:19.935457 8946 ts_tablet_manager.cc:1367] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.113s user 0.058s sys 0.021s
I20240501 20:11:19.937822 8946 raft_consensus.cc:348] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:19.938125 8946 raft_consensus.cc:729] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:19.938522 8946 consensus_queue.cc:260] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:19.944221 8946 ts_tablet_manager.cc:1398] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240501 20:11:19.945286 4337 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-2e97fe7ab97a4458b5e977af3d1356df received from {username='slave'} at 127.28.136.71:52891
I20240501 20:11:19.945433 4337 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session 9592068f16ca406eb8a347a65d365782-2e97fe7ab97a4458b5e977af3d1356df on tablet 2e97fe7ab97a4458b5e977af3d1356df with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:19.964066 8981 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:19.964540 8981 tablet_bootstrap.cc:492] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:19.964860 8981 ts_tablet_manager.cc:1367] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.094s user 0.044s sys 0.009s
I20240501 20:11:19.965675 8981 raft_consensus.cc:348] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:19.965947 8981 raft_consensus.cc:729] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:19.966156 8981 consensus_queue.cc:260] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:19.966765 8981 ts_tablet_manager.cc:1398] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:19.967649 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-96005b2bfa9847bb992640ad40a68b36 received from {username='slave'} at 127.28.136.71:50981
I20240501 20:11:19.967792 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session 9592068f16ca406eb8a347a65d365782-96005b2bfa9847bb992640ad40a68b36 on tablet 96005b2bfa9847bb992640ad40a68b36 with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:19.968047 8968 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:19.969010 8968 tablet_bootstrap.cc:492] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: Bootstrap complete.
I20240501 20:11:19.969331 8968 ts_tablet_manager.cc:1367] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent bootstrapping tablet: real 0.115s user 0.055s sys 0.014s
I20240501 20:11:19.970141 8968 raft_consensus.cc:348] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:19.970415 8968 raft_consensus.cc:729] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Initialized, Role: LEARNER
I20240501 20:11:19.970604 8968 consensus_queue.cc:260] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } }
I20240501 20:11:19.971174 8968 ts_tablet_manager.cc:1398] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:19.971869 4599 tablet_copy_service.cc:342] P 75b5acf03b9f4f6e98a67ae046587e79: Request end of tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-d48a2b58ce2d417e9abf63c292419d14 received from {username='slave'} at 127.28.136.69:54795
I20240501 20:11:19.972005 4599 tablet_copy_service.cc:433] P 75b5acf03b9f4f6e98a67ae046587e79: ending tablet copy session ee4daecd449e4fc981fd1fe2b0e492c2-d48a2b58ce2d417e9abf63c292419d14 on tablet d48a2b58ce2d417e9abf63c292419d14 with peer ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:19.973531 9021 tablet_copy_client.cc:523] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:19.977604 9021 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:20.016594 9021 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.017174 9021 tablet_bootstrap.cc:492] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:20.017518 9021 ts_tablet_manager.cc:1367] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.040s user 0.035s sys 0.005s
I20240501 20:11:20.018373 9021 raft_consensus.cc:348] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:20.018683 9021 raft_consensus.cc:729] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:20.018899 9021 consensus_queue.cc:260] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:20.019520 9021 ts_tablet_manager.cc:1398] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:20.020350 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-d09b0474ecbf4d458d7a520c9e9f329b received from {username='slave'} at 127.28.136.70:39057
I20240501 20:11:20.020493 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-d09b0474ecbf4d458d7a520c9e9f329b on tablet d09b0474ecbf4d458d7a520c9e9f329b with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:20.040633 7467 raft_consensus.cc:1203] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.047318 9026 tablet_copy_client.cc:523] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:20.050422 9026 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:20.066650 9031 tablet_copy_client.cc:523] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:20.070036 9031 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:20.078778 9036 tablet_copy_client.cc:523] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:20.082298 9036 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap starting.
I20240501 20:11:20.085345 6547 raft_consensus.cc:1203] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:11:20.115501 7467 raft_consensus.cc:1203] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.120790 9026 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.121328 9026 tablet_bootstrap.cc:492] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:20.121627 9026 ts_tablet_manager.cc:1367] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.071s user 0.045s sys 0.005s
I20240501 20:11:20.123054 9026 raft_consensus.cc:348] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:20.123360 9026 raft_consensus.cc:729] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:20.123585 9026 consensus_queue.cc:260] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:20.126745 9026 ts_tablet_manager.cc:1398] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:11:20.129624 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-7d320136fd1d4eb895362144ef6a5322 received from {username='slave'} at 127.28.136.73:53095
I20240501 20:11:20.129745 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-7d320136fd1d4eb895362144ef6a5322 on tablet 7d320136fd1d4eb895362144ef6a5322 with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:20.132339 9031 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.132721 9031 tablet_bootstrap.cc:492] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:20.133013 9031 ts_tablet_manager.cc:1367] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.063s user 0.038s sys 0.010s
I20240501 20:11:20.133622 9031 raft_consensus.cc:348] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:20.133864 9031 raft_consensus.cc:729] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:20.134061 9031 consensus_queue.cc:260] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:20.134625 9031 ts_tablet_manager.cc:1398] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:20.135530 4337 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-11f240637b1e4ef8bc3414e11b38f469 received from {username='slave'} at 127.28.136.71:52891
I20240501 20:11:20.135668 4337 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session 9592068f16ca406eb8a347a65d365782-11f240637b1e4ef8bc3414e11b38f469 on tablet 11f240637b1e4ef8bc3414e11b38f469 with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:20.138289 9041 tablet_copy_client.cc:523] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:20.140537 9036 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.140974 9036 tablet_bootstrap.cc:492] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: Bootstrap complete.
I20240501 20:11:20.141242 9036 ts_tablet_manager.cc:1367] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: Time spent bootstrapping tablet: real 0.059s user 0.041s sys 0.010s
I20240501 20:11:20.141867 9041 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:20.142194 9046 tablet_copy_client.cc:523] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:20.142060 9036 raft_consensus.cc:348] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:20.142315 9036 raft_consensus.cc:729] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55fe2a4464a9402f927b2b60609b34dd, State: Initialized, Role: LEARNER
I20240501 20:11:20.142524 9036 consensus_queue.cc:260] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: true } }
I20240501 20:11:20.143086 9036 ts_tablet_manager.cc:1398] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:20.143831 6282 raft_consensus.cc:1203] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.144927 9046 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:20.147414 4337 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session 55fe2a4464a9402f927b2b60609b34dd-79cb3f8b3fe94497bb54d3d8b7f18754 received from {username='slave'} at 127.28.136.72:50629
I20240501 20:11:20.147552 4337 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session 55fe2a4464a9402f927b2b60609b34dd-79cb3f8b3fe94497bb54d3d8b7f18754 on tablet 79cb3f8b3fe94497bb54d3d8b7f18754 with peer 55fe2a4464a9402f927b2b60609b34dd
I20240501 20:11:20.178907 6416 raft_consensus.cc:1203] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.196913 6147 raft_consensus.cc:1203] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.200804 9046 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.201133 9041 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.201287 9046 tablet_bootstrap.cc:492] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:20.201516 9041 tablet_bootstrap.cc:492] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:20.201614 9046 ts_tablet_manager.cc:1367] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.057s user 0.046s sys 0.007s
I20240501 20:11:20.201812 9041 ts_tablet_manager.cc:1367] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.060s user 0.047s sys 0.004s
I20240501 20:11:20.202538 9041 raft_consensus.cc:348] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:20.202823 9041 raft_consensus.cc:729] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:20.202670 9046 raft_consensus.cc:348] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:20.202962 9046 raft_consensus.cc:729] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:20.203014 9041 consensus_queue.cc:260] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:20.203140 9046 consensus_queue.cc:260] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:20.203588 9041 ts_tablet_manager.cc:1398] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:20.203753 9046 ts_tablet_manager.cc:1398] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:20.204963 4337 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-0822a3fc82134d018bae96fecc62fdd3 received from {username='slave'} at 127.28.136.70:48699
I20240501 20:11:20.205091 4337 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-0822a3fc82134d018bae96fecc62fdd3 on tablet 0822a3fc82134d018bae96fecc62fdd3 with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:20.206187 4337 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-eeb1eec290ce4dcbb2afedd57e3b81da received from {username='slave'} at 127.28.136.73:45225
I20240501 20:11:20.206293 4337 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-eeb1eec290ce4dcbb2afedd57e3b81da on tablet eeb1eec290ce4dcbb2afedd57e3b81da with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:20.262647 6282 raft_consensus.cc:1203] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.314754 9122 tablet_copy_client.cc:523] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:20.317478 9122 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:20.342967 9126 tablet_copy_client.cc:523] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:20.346678 9126 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: Bootstrap starting.
I20240501 20:11:20.348780 9131 tablet_copy_client.cc:523] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:20.351347 9131 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: Bootstrap starting.
I20240501 20:11:20.352895 9122 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.353451 9122 tablet_bootstrap.cc:492] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:20.353734 9122 ts_tablet_manager.cc:1367] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.036s user 0.034s sys 0.002s
I20240501 20:11:20.354593 9122 raft_consensus.cc:348] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:20.354866 9122 raft_consensus.cc:729] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:20.355082 9122 consensus_queue.cc:260] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:20.355801 9122 ts_tablet_manager.cc:1398] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:20.356725 6303 tablet_copy_service.cc:342] P f5195730e32a4abd8519baf9961d32f2: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-3f7fb3f245414d05a8e5d837076195c2 received from {username='slave'} at 127.28.136.73:33971
I20240501 20:11:20.356953 6303 tablet_copy_service.cc:433] P f5195730e32a4abd8519baf9961d32f2: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-3f7fb3f245414d05a8e5d837076195c2 on tablet 3f7fb3f245414d05a8e5d837076195c2 with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:20.373206 7467 raft_consensus.cc:1203] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.391850 6416 raft_consensus.cc:1203] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.392331 9131 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.392869 9131 tablet_bootstrap.cc:492] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: Bootstrap complete.
I20240501 20:11:20.393152 9131 ts_tablet_manager.cc:1367] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: Time spent bootstrapping tablet: real 0.042s user 0.035s sys 0.006s
I20240501 20:11:20.393997 9131 raft_consensus.cc:348] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:20.394286 9131 raft_consensus.cc:729] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5195730e32a4abd8519baf9961d32f2, State: Initialized, Role: LEARNER
I20240501 20:11:20.394487 9131 consensus_queue.cc:260] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: true } }
I20240501 20:11:20.395092 9131 ts_tablet_manager.cc:1398] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:20.395931 4337 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session f5195730e32a4abd8519baf9961d32f2-c11109bc41e34ab58fe3c9f568adf85e received from {username='slave'} at 127.28.136.70:48699
I20240501 20:11:20.396067 4337 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session f5195730e32a4abd8519baf9961d32f2-c11109bc41e34ab58fe3c9f568adf85e on tablet c11109bc41e34ab58fe3c9f568adf85e with peer f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:20.396648 9126 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:20.397018 9126 tablet_bootstrap.cc:492] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: Bootstrap complete.
I20240501 20:11:20.397243 9126 ts_tablet_manager.cc:1367] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: Time spent bootstrapping tablet: real 0.051s user 0.049s sys 0.000s
I20240501 20:11:20.397974 9126 raft_consensus.cc:348] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:20.398234 9126 raft_consensus.cc:729] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Initialized, Role: LEARNER
I20240501 20:11:20.398434 9126 consensus_queue.cc:260] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } }
I20240501 20:11:20.399148 9126 ts_tablet_manager.cc:1398] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:20.399924 4599 tablet_copy_service.cc:342] P 75b5acf03b9f4f6e98a67ae046587e79: Request end of tablet copy session 9592068f16ca406eb8a347a65d365782-ca4d34ed427e44ed88379eb69feaeb8a received from {username='slave'} at 127.28.136.71:44331
I20240501 20:11:20.400089 4599 tablet_copy_service.cc:433] P 75b5acf03b9f4f6e98a67ae046587e79: ending tablet copy session 9592068f16ca406eb8a347a65d365782-ca4d34ed427e44ed88379eb69feaeb8a on tablet ca4d34ed427e44ed88379eb69feaeb8a with peer 9592068f16ca406eb8a347a65d365782
I20240501 20:11:20.403750 7467 raft_consensus.cc:1203] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.435482 6416 raft_consensus.cc:1203] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.501044 6547 raft_consensus.cc:1203] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.510221 9453 raft_consensus.cc:1050] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:20.511065 9453 consensus_queue.cc:237] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:20.513727 7467 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.514492 9452 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.514822 4578 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.515306 6416 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.515736 9453 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.516263 9232 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.523533 8769 raft_consensus.cc:1050] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:20.524492 8769 consensus_queue.cc:237] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:20.527334 6147 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.527341 7466 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.527953 8868 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.528546 8868 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.537196 9456 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.538039 6416 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.542356 4578 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.543742 4032 catalog_manager.cc:5484] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:20.566226 4578 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.566800 8868 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.569183 7466 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.570262 8868 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.575906 6147 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.579821 4577 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.582383 8769 raft_consensus.cc:1050] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:20.583449 8769 consensus_queue.cc:237] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:20.585916 4029 catalog_manager.cc:5484] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.593220 6416 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.594175 6282 raft_consensus.cc:1203] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.594108 8868 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.598393 7467 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.605715 4449 consensus_queue.cc:237] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:20.608382 6415 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:20.609525 9452 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:20.610953 4577 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.611673 9474 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.613929 9452 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.615518 6147 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.616251 9474 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.617617 6415 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.621528 7466 raft_consensus.cc:1203] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:11:20.624893 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb36776ab8b74586aead4ecad56cc815 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:20.628296 7467 raft_consensus.cc:1261] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:20.628387 4030 catalog_manager.cc:5484] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:20.628918 9452 consensus_queue.cc:1035] T bb36776ab8b74586aead4ecad56cc815 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:20.630582 7466 raft_consensus.cc:2932] T bb36776ab8b74586aead4ecad56cc815 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.632761 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet bb36776ab8b74586aead4ecad56cc815 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:20.644977 9482 tablet_replica.cc:330] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:20.645205 9482 raft_consensus.cc:2229] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:20.646319 9482 raft_consensus.cc:2258] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:20.647751 9482 ts_tablet_manager.cc:1875] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:20.654198 9482 ts_tablet_manager.cc:1888] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:20.654650 9482 log.cc:1196] T bb36776ab8b74586aead4ecad56cc815 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb36776ab8b74586aead4ecad56cc815
I20240501 20:11:20.654522 8868 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:20.655594 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet bb36776ab8b74586aead4ecad56cc815 (table rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]) successfully deleted
I20240501 20:11:20.660518 6416 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:20.657568 4577 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:20.660576 6148 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:20.664532 4030 catalog_manager.cc:5484] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:20.669436 9481 raft_consensus.cc:1050] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:20.669672 4318 consensus_queue.cc:237] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:20.671572 6148 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:20.672086 6416 raft_consensus.cc:1261] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:20.672109 8868 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:20.672896 9481 consensus_queue.cc:237] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:20.674281 8868 consensus_queue.cc:1035] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:20.675429 8868 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:20.678345 6547 raft_consensus.cc:1261] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:20.678987 6148 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:20.679071 6416 raft_consensus.cc:2932] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:20.679746 4318 raft_consensus.cc:1261] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:20.680037 4185 raft_consensus.cc:1261] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:11:20.680313 9486 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:20.680666 8501 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:20.683115 9481 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:20.685030 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11f240637b1e4ef8bc3414e11b38f469 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:20.686825 4030 catalog_manager.cc:5484] T 11f240637b1e4ef8bc3414e11b38f469 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:20.692135 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 11f240637b1e4ef8bc3414e11b38f469 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:20.692566 9482 tablet_replica.cc:330] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:20.692811 9482 raft_consensus.cc:2229] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:20.693065 9482 raft_consensus.cc:2258] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:20.694123 9482 ts_tablet_manager.cc:1875] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:20.701277 9486 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:20.702251 9482 ts_tablet_manager.cc:1888] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:20.702370 9482 log.cc:1196] T 11f240637b1e4ef8bc3414e11b38f469 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/11f240637b1e4ef8bc3414e11b38f469
I20240501 20:11:20.702872 6546 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:20.705154 4185 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:20.709403 4030 catalog_manager.cc:5484] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:20.708632 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 11f240637b1e4ef8bc3414e11b38f469 (table rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]) successfully deleted
I20240501 20:11:20.718591 4318 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:20.720412 9499 raft_consensus.cc:1050] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:20.722024 9499 consensus_queue.cc:237] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:20.722352 4577 consensus_queue.cc:237] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:20.724277 6546 raft_consensus.cc:1261] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:20.724880 9486 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:20.727104 4318 raft_consensus.cc:1261] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:11:20.727663 9486 consensus_queue.cc:1035] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:20.729493 8501 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:20.730423 6546 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:20.733042 6148 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.733940 4449 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.734925 4318 raft_consensus.cc:2932] T d1e01586b0ba42869481577ceacc84e8 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:20.736115 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1e01586b0ba42869481577ceacc84e8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:20.738215 9133 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.738600 6282 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.738866 9133 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.739416 9133 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.738893 4035 catalog_manager.cc:5484] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 5 to 6, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:20.753268 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e01586b0ba42869481577ceacc84e8 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:20.754050 9510 tablet_replica.cc:330] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:20.755301 8501 raft_consensus.cc:1050] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:20.756165 8501 consensus_queue.cc:237] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:20.756632 9510 raft_consensus.cc:2229] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:20.757020 9510 raft_consensus.cc:2258] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:20.758006 9510 ts_tablet_manager.cc:1875] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:20.759920 4447 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.760063 6281 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.760643 9486 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.763373 6148 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.763883 8501 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.764258 9486 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.765265 9510 ts_tablet_manager.cc:1888] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:11:20.765446 9510 log.cc:1196] T d1e01586b0ba42869481577ceacc84e8 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/d1e01586b0ba42869481577ceacc84e8
I20240501 20:11:20.766443 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet d1e01586b0ba42869481577ceacc84e8 (table rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]) successfully deleted
I20240501 20:11:20.767376 8501 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.769819 9481 raft_consensus.cc:1050] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79: attempting to promote NON_VOTER ee4daecd449e4fc981fd1fe2b0e492c2 to VOTER
I20240501 20:11:20.770668 9481 consensus_queue.cc:237] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:20.769068 6148 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.772763 4447 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.776829 4035 catalog_manager.cc:5484] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.781860 6281 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.785982 4577 consensus_queue.cc:237] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:20.789111 6147 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 LEARNER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.789397 6416 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.789660 9517 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.790060 9481 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.790242 6281 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:20.790745 9481 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:20.794057 4318 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.795244 9517 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.797643 6148 raft_consensus.cc:1261] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:20.798183 9517 consensus_queue.cc:1035] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:20.800526 9481 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.801311 6146 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.805007 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae0fe7a1daab476391d80406766b3b11 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:20.808022 4030 catalog_manager.cc:5484] T ae0fe7a1daab476391d80406766b3b11 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:20.808344 6280 raft_consensus.cc:2932] T ae0fe7a1daab476391d80406766b3b11 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.814920 9474 raft_consensus.cc:1050] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:20.815865 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet ae0fe7a1daab476391d80406766b3b11 with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:20.815845 9474 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:20.823794 9499 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.826215 9133 raft_consensus.cc:1050] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:20.827221 9133 consensus_queue.cc:237] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:20.829622 4447 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.838128 9503 raft_consensus.cc:1050] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:20.838663 4035 catalog_manager.cc:5484] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.838996 9503 consensus_queue.cc:237] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:20.840237 6147 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.845973 6281 raft_consensus.cc:1203] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:20.866267 6415 raft_consensus.cc:1203] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:11:20.866750 7467 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.867403 8868 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.867761 4185 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.867965 6546 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.868331 8868 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.868927 8868 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.878993 6282 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:20.882017 9481 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:20.887560 6415 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:20.888195 4035 catalog_manager.cc:5484] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 7 to 8, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:20.895768 4577 consensus_queue.cc:237] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } }
I20240501 20:11:20.896406 6147 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:20.903028 6414 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:20.904151 9481 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:20.905561 7466 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.906199 9498 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.913986 9471 raft_consensus.cc:1050] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:20.914808 9471 consensus_queue.cc:237] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:20.921348 9481 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:20.927038 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d48a2b58ce2d417e9abf63c292419d14 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:20.927871 6415 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:20.927349 8868 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.932600 4035 catalog_manager.cc:5484] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 8 to 9, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:20.937741 9523 tablet_replica.cc:330] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:20.938050 9523 raft_consensus.cc:2229] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:20.938544 9523 raft_consensus.cc:2258] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:20.939158 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet d48a2b58ce2d417e9abf63c292419d14 with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:20.939694 9523 ts_tablet_manager.cc:1875] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:20.941478 6282 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.941291 7466 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.943235 9133 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.943886 4030 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:20.952010 4449 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.952678 9133 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.957055 4185 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.960012 9533 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.966599 6281 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.967638 7466 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.970748 6546 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.972592 4030 catalog_manager.cc:5484] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.975971 9523 ts_tablet_manager.cc:1888] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:20.976161 9523 log.cc:1196] T ae0fe7a1daab476391d80406766b3b11 P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae0fe7a1daab476391d80406766b3b11
I20240501 20:11:20.976651 6146 raft_consensus.cc:1261] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:20.977442 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet ae0fe7a1daab476391d80406766b3b11 (table rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]) successfully deleted
I20240501 20:11:20.977764 9481 consensus_queue.cc:1035] T d48a2b58ce2d417e9abf63c292419d14 P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:20.979338 6146 raft_consensus.cc:2932] T d48a2b58ce2d417e9abf63c292419d14 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } }
I20240501 20:11:20.983454 4447 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:20.995573 6415 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.996299 9531 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.997924 6416 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.998852 4577 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.998965 9463 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:20.999208 6282 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:20.999645 9463 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.001943 8769 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.002934 4186 consensus_queue.cc:237] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.005263 6281 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.006019 9533 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.009496 9133 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.010087 6281 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.013509 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d320136fd1d4eb895362144ef6a5322 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:21.017289 4035 catalog_manager.cc:5484] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:21.023288 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d320136fd1d4eb895362144ef6a5322 with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:21.023699 9523 tablet_replica.cc:330] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:21.023906 9523 raft_consensus.cc:2229] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.024142 9523 raft_consensus.cc:2258] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.025548 9523 ts_tablet_manager.cc:1875] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.031039 6147 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.031617 9533 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.032300 9523 ts_tablet_manager.cc:1888] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:21.032518 9523 log.cc:1196] T 7d320136fd1d4eb895362144ef6a5322 P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/7d320136fd1d4eb895362144ef6a5322
I20240501 20:11:21.033999 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 7d320136fd1d4eb895362144ef6a5322 (table rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]) successfully deleted
I20240501 20:11:21.035521 4447 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.036223 9503 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.039990 9533 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.045363 6147 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.047444 4030 catalog_manager.cc:5484] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:21.053138 9474 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.054028 4186 consensus_queue.cc:237] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:21.045745 6415 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.061828 6282 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.065336 6416 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.066334 9533 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.069622 9539 tablet_replica.cc:330] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:21.069813 9539 raft_consensus.cc:2229] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.069983 9539 pending_rounds.cc:62] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: Trying to abort 1 pending ops.
I20240501 20:11:21.069423 9503 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.070065 9539 pending_rounds.cc:68] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022978174034239488 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d48a2b58ce2d417e9abf63c292419d14" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: NON_VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } } }
I20240501 20:11:21.070437 9539 raft_consensus.cc:2894] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 20:11:21.070535 9539 raft_consensus.cc:2866] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:11:21.070618 9539 raft_consensus.cc:2258] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.077430 4030 catalog_manager.cc:5484] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 reported cstate change: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.078495 7467 raft_consensus.cc:1261] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.080921 9533 consensus_queue.cc:1035] T 7d320136fd1d4eb895362144ef6a5322 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.083685 4577 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.084646 9539 ts_tablet_manager.cc:1875] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.085184 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96005b2bfa9847bb992640ad40a68b36 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:21.085083 6416 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.089023 7466 raft_consensus.cc:2932] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.091938 4026 catalog_manager.cc:5484] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.093907 6146 raft_consensus.cc:1261] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.094777 9499 consensus_queue.cc:1035] T 96005b2bfa9847bb992640ad40a68b36 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.095788 9471 raft_consensus.cc:1050] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER 55fe2a4464a9402f927b2b60609b34dd to VOTER
I20240501 20:11:21.096719 9471 consensus_queue.cc:237] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:21.096621 6146 raft_consensus.cc:2932] T 96005b2bfa9847bb992640ad40a68b36 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.099506 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 96005b2bfa9847bb992640ad40a68b36 with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:21.110538 9523 tablet_replica.cc:330] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:21.110883 9523 raft_consensus.cc:2229] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.111693 9523 pending_rounds.cc:62] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: Trying to abort 1 pending ops.
I20240501 20:11:21.111888 9523 pending_rounds.cc:68] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022978174314307584 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "96005b2bfa9847bb992640ad40a68b36" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: NON_VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } } }
I20240501 20:11:21.112483 9523 raft_consensus.cc:2894] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 20:11:21.112658 9523 raft_consensus.cc:2866] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:11:21.112843 9523 raft_consensus.cc:2258] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.113556 6415 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.116530 8868 raft_consensus.cc:1050] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:21.117484 8868 consensus_queue.cc:237] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:21.120527 4318 consensus_queue.cc:237] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:21.123386 6282 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.124058 9463 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.125841 9523 ts_tablet_manager.cc:1875] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.127002 9463 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.130405 6416 raft_consensus.cc:1261] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.131374 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e97fe7ab97a4458b5e977af3d1356df with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:21.131577 6281 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.135607 9474 consensus_queue.cc:1035] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.140748 6415 raft_consensus.cc:2932] T 2e97fe7ab97a4458b5e977af3d1356df P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.142932 6281 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.143602 9474 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.146718 6147 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.147267 9539 ts_tablet_manager.cc:1888] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:21.147490 9539 log.cc:1196] T d48a2b58ce2d417e9abf63c292419d14 P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/d48a2b58ce2d417e9abf63c292419d14
I20240501 20:11:21.147392 9474 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.148576 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet d48a2b58ce2d417e9abf63c292419d14 (table rebalance_test_table_3 [id=332a54d306944b40be9a6cf83744d35c]) successfully deleted
I20240501 20:11:21.151093 4185 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.151858 8769 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.153105 9523 ts_tablet_manager.cc:1888] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:21.153393 9523 log.cc:1196] T 96005b2bfa9847bb992640ad40a68b36 P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/96005b2bfa9847bb992640ad40a68b36
I20240501 20:11:21.154500 6416 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.154309 4026 catalog_manager.cc:5484] T 2e97fe7ab97a4458b5e977af3d1356df P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:21.154866 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 96005b2bfa9847bb992640ad40a68b36 (table rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]) successfully deleted
I20240501 20:11:21.155576 9474 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.156332 4447 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.157583 8769 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.158083 6546 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.158752 9471 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.159194 9476 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.161125 6282 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.164395 4185 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.165567 6147 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.166792 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e97fe7ab97a4458b5e977af3d1356df with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:21.167192 9482 tablet_replica.cc:330] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:21.167402 9482 raft_consensus.cc:2229] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.167779 9482 raft_consensus.cc:2258] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.169036 9482 ts_tablet_manager.cc:1875] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.173871 4035 catalog_manager.cc:5484] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.181548 9463 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:21.182776 6416 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:21.184245 4318 consensus_queue.cc:237] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:21.186599 6546 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:21.187037 6146 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.187638 8868 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.188674 6282 raft_consensus.cc:1261] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.191294 8769 consensus_queue.cc:1035] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.192538 8769 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.192708 4026 catalog_manager.cc:5484] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 7 to 8, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:21.192690 6147 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.198448 4026 catalog_manager.cc:5484] T 0822a3fc82134d018bae96fecc62fdd3 P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 8 to 9, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.200412 4447 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:21.201532 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0822a3fc82134d018bae96fecc62fdd3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:21.203002 6281 raft_consensus.cc:2932] T 0822a3fc82134d018bae96fecc62fdd3 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.199595 9482 ts_tablet_manager.cc:1888] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:21.206245 9482 log.cc:1196] T 2e97fe7ab97a4458b5e977af3d1356df P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/2e97fe7ab97a4458b5e977af3d1356df
I20240501 20:11:21.206791 4318 consensus_queue.cc:237] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:21.207955 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet 0822a3fc82134d018bae96fecc62fdd3 with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:21.208410 9510 tablet_replica.cc:330] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:21.208628 9510 raft_consensus.cc:2229] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.208900 9510 raft_consensus.cc:2258] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.208974 6416 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.210269 9510 ts_tablet_manager.cc:1875] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.210188 8769 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.211216 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 2e97fe7ab97a4458b5e977af3d1356df (table rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]) successfully deleted
I20240501 20:11:21.211743 6546 raft_consensus.cc:1261] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.212366 8868 consensus_queue.cc:1035] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.213268 8868 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:21.216830 9510 ts_tablet_manager.cc:1888] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:21.217007 9510 log.cc:1196] T 0822a3fc82134d018bae96fecc62fdd3 P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/0822a3fc82134d018bae96fecc62fdd3
I20240501 20:11:21.217816 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet 0822a3fc82134d018bae96fecc62fdd3 (table rebalance_test_table_6 [id=c87ce157f6e54d03a722dbdad1cb1379]) successfully deleted
I20240501 20:11:21.217788 6416 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:21.218634 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:21.221020 6547 raft_consensus.cc:2932] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } }
I20240501 20:11:21.224838 4035 catalog_manager.cc:5484] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:21.230185 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 79cb3f8b3fe94497bb54d3d8b7f18754 with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:21.230544 9523 tablet_replica.cc:330] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:21.230747 9523 raft_consensus.cc:2229] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.230983 9523 raft_consensus.cc:2258] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.232322 9523 ts_tablet_manager.cc:1875] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.240979 9523 ts_tablet_manager.cc:1888] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:21.241134 9523 log.cc:1196] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/79cb3f8b3fe94497bb54d3d8b7f18754
I20240501 20:11:21.241395 9234 raft_consensus.cc:1050] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:21.242429 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 79cb3f8b3fe94497bb54d3d8b7f18754 (table rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]) successfully deleted
I20240501 20:11:21.242355 9234 consensus_queue.cc:237] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.245833 4318 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.246135 4577 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.246469 9571 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.246569 7466 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12 [term 2 LEARNER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.247327 9571 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.247946 9569 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.253465 9571 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.258114 4577 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.258724 7466 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.260046 4030 catalog_manager.cc:5484] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 reported cstate change: config changed from index 8 to 9, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f5195730e32a4abd8519baf9961d32f2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:21.262948 4318 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.271329 6281 consensus_queue.cc:237] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.273228 7466 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:21.273339 4318 raft_consensus.cc:1261] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer f5195730e32a4abd8519baf9961d32f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:21.273772 9569 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:21.274312 9234 consensus_queue.cc:1035] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:21.275967 9569 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.278261 4318 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.282050 7466 raft_consensus.cc:2932] T 3f7fb3f245414d05a8e5d837076195c2 P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.282434 4019 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f7fb3f245414d05a8e5d837076195c2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:21.283443 4031 catalog_manager.cc:5484] T 3f7fb3f245414d05a8e5d837076195c2 P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 9 to 10, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 2 leader_uuid: "f5195730e32a4abd8519baf9961d32f2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:21.289062 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f7fb3f245414d05a8e5d837076195c2 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:21.289435 9482 tablet_replica.cc:330] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:21.289649 9482 raft_consensus.cc:2229] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.290153 9482 raft_consensus.cc:2258] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.291059 9482 ts_tablet_manager.cc:1875] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.293043 8769 raft_consensus.cc:1050] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER f5195730e32a4abd8519baf9961d32f2 to VOTER
I20240501 20:11:21.293926 8769 consensus_queue.cc:237] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:21.296912 6281 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.297494 6547 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.297442 8868 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.298107 4185 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:21.298913 9482 ts_tablet_manager.cc:1888] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:11:21.299039 9482 log.cc:1196] T 3f7fb3f245414d05a8e5d837076195c2 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/3f7fb3f245414d05a8e5d837076195c2
I20240501 20:11:21.299014 9463 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.299564 9463 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:21.299805 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 3f7fb3f245414d05a8e5d837076195c2 (table rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]) successfully deleted
I20240501 20:11:21.315121 9463 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.315958 6281 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.316556 6547 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.322033 4031 catalog_manager.cc:5484] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 reported cstate change: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.323733 4185 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5195730e32a4abd8519baf9961d32f2 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.331521 4318 consensus_queue.cc:237] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:21.333922 6281 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.334663 6547 raft_consensus.cc:1261] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:21.334702 8868 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.334949 4318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb"
dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53698
I20240501 20:11:21.335161 4318 raft_consensus.cc:595] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:21.335232 9474 consensus_queue.cc:1035] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.339043 8868 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.339776 6547 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.342442 6281 raft_consensus.cc:2932] T c11109bc41e34ab58fe3c9f568adf85e P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:21.343573 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c11109bc41e34ab58fe3c9f568adf85e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:21.345815 4030 catalog_manager.cc:5484] T c11109bc41e34ab58fe3c9f568adf85e P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 8 to 9, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:21.350833 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet c11109bc41e34ab58fe3c9f568adf85e with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:21.351235 9510 tablet_replica.cc:330] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:21.351472 9510 raft_consensus.cc:2229] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.351858 9510 raft_consensus.cc:2258] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.352888 9510 ts_tablet_manager.cc:1875] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.355187 4185 consensus_queue.cc:237] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:21.357429 6416 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:21.357728 4118 consensus_peers.cc:485] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 618ea2abb259425babf56375f41eb81c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:21.358070 4577 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:21.358112 9533 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:21.358845 9533 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:21.360793 9499 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.361852 4577 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.362277 9510 ts_tablet_manager.cc:1888] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:21.362442 9510 log.cc:1196] T c11109bc41e34ab58fe3c9f568adf85e P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/c11109bc41e34ab58fe3c9f568adf85e
I20240501 20:11:21.363205 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet c11109bc41e34ab58fe3c9f568adf85e (table rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]) successfully deleted
I20240501 20:11:21.365537 6416 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.368978 4026 catalog_manager.cc:5484] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:21.431226 8868 raft_consensus.cc:979] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: : Instructing follower 75b5acf03b9f4f6e98a67ae046587e79 to start an election
I20240501 20:11:21.431382 8868 raft_consensus.cc:1067] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Signalling peer 75b5acf03b9f4f6e98a67ae046587e79 to start an election
I20240501 20:11:21.432121 4577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb"
dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
from {username='slave'} at 127.28.136.66:54739
I20240501 20:11:21.432300 4577 raft_consensus.cc:482] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:21.432417 4577 raft_consensus.cc:3037] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.435201 4577 raft_consensus.cc:504] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.437172 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:21.437394 4318 raft_consensus.cc:3032] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:21.437489 4318 raft_consensus.cc:729] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:11:21.437762 4318 consensus_queue.cc:260] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.438259 4318 raft_consensus.cc:3037] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.441352 4318 raft_consensus.cc:2454] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 2.
I20240501 20:11:21.442639 6147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2"
I20240501 20:11:21.442858 6147 raft_consensus.cc:3037] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.445647 6147 raft_consensus.cc:2454] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 2.
I20240501 20:11:21.447010 4513 leader_election.cc:302] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, 75b5acf03b9f4f6e98a67ae046587e79, ee4daecd449e4fc981fd1fe2b0e492c2; no voters:
I20240501 20:11:21.447338 8501 raft_consensus.cc:2781] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:21.447490 8501 raft_consensus.cc:686] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 LEADER]: Becoming Leader. State: Replica: 75b5acf03b9f4f6e98a67ae046587e79, State: Running, Role: LEADER
I20240501 20:11:21.447809 8501 consensus_queue.cc:237] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.448873 4577 leader_election.cc:288] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [CANDIDATE]: Term 2 election: Requested vote from peers 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155), b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209)
I20240501 20:11:21.453302 9517 raft_consensus.cc:1050] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79: attempting to promote NON_VOTER 9592068f16ca406eb8a347a65d365782 to VOTER
I20240501 20:11:21.454195 9517 consensus_queue.cc:237] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:21.457671 6416 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782 [term 2 LEARNER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.458299 6147 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.459074 9481 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.459414 8501 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.460624 4030 catalog_manager.cc:5484] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: term changed from 1 to 2, leader changed from 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) to 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68). New cstate: current_term: 2 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:21.464120 4318 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.464821 8501 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.468827 8501 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.473408 6147 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.473446 6416 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.480101 4031 catalog_manager.cc:5484] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 8 to 9, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.483340 4318 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9592068f16ca406eb8a347a65d365782 (127.28.136.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.486327 4577 consensus_queue.cc:237] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:21.486745 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdac8c0b92d442ebd84fb43af7e32fb" candidate_uuid: "75b5acf03b9f4f6e98a67ae046587e79" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b662c3da2b1b4f3693920ccb680b8d12"
I20240501 20:11:21.487071 7466 raft_consensus.cc:3037] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.489406 6147 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:21.490067 7466 raft_consensus.cc:2454] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b5acf03b9f4f6e98a67ae046587e79 in term 2.
I20240501 20:11:21.490319 9517 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:21.493616 6416 raft_consensus.cc:1261] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:21.494733 9517 consensus_queue.cc:1035] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:21.497186 9517 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.501039 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca4d34ed427e44ed88379eb69feaeb8a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:21.503155 6147 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.504133 4030 catalog_manager.cc:5484] T ca4d34ed427e44ed88379eb69feaeb8a P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 9 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 2 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:21.504808 6416 raft_consensus.cc:2932] T ca4d34ed427e44ed88379eb69feaeb8a P 9592068f16ca406eb8a347a65d365782 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } }
I20240501 20:11:21.514650 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4d34ed427e44ed88379eb69feaeb8a with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:21.515142 9539 tablet_replica.cc:330] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:21.515388 9539 raft_consensus.cc:2229] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:21.515651 9539 raft_consensus.cc:2258] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:21.519274 9539 ts_tablet_manager.cc:1875] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:21.526307 9539 ts_tablet_manager.cc:1888] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:11:21.526535 9539 log.cc:1196] T ca4d34ed427e44ed88379eb69feaeb8a P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/ca4d34ed427e44ed88379eb69feaeb8a
I20240501 20:11:21.527437 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet ca4d34ed427e44ed88379eb69feaeb8a (table rebalance_test_table_2 [id=b1531cc70fcf432e8abb13b6734bc43b]) successfully deleted
I20240501 20:11:21.555450 4318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da"
dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53752
I20240501 20:11:21.555683 4318 raft_consensus.cc:595] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:21.583379 4185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b"
dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32874
I20240501 20:11:21.583595 4185 raft_consensus.cc:595] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:21.613433 4185 consensus_queue.cc:237] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:21.615775 4447 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:21.615798 4118 consensus_peers.cc:485] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d7b1a47b1fe4e4b9a60b18b0b99056b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:21.615782 6416 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:21.616305 9499 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:21.616889 9503 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:21.619532 9533 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.620186 4447 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.620280 6416 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.624619 4026 catalog_manager.cc:5484] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:21.644526 4447 consensus_queue.cc:237] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:21.647194 4577 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:21.647393 4384 consensus_peers.cc:485] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3b4a011c5944a1db9dbc496e87d5768. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:21.647444 6281 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:21.647933 9452 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:21.648211 9456 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:21.651072 9452 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.651732 4577 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.656592 6281 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:21.657490 4030 catalog_manager.cc:5484] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:21.718942 4318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da"
dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53772
I20240501 20:11:21.719192 4318 raft_consensus.cc:595] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:21.735157 4185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b"
dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32922
I20240501 20:11:21.735350 4185 raft_consensus.cc:595] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:21.773058 9503 raft_consensus.cc:979] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: : Instructing follower ee4daecd449e4fc981fd1fe2b0e492c2 to start an election
I20240501 20:11:21.773239 9503 raft_consensus.cc:1067] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Signalling peer ee4daecd449e4fc981fd1fe2b0e492c2 to start an election
I20240501 20:11:21.773950 6147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b"
dest_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2"
from {username='slave'} at 127.28.136.65:40047
I20240501 20:11:21.774188 6147 raft_consensus.cc:482] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:21.774309 6147 raft_consensus.cc:3037] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.776892 6147 raft_consensus.cc:504] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:21.778249 6147 leader_election.cc:288] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [CANDIDATE]: Term 2 election: Requested vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983)
I20240501 20:11:21.778579 4185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b" candidate_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
I20240501 20:11:21.778806 4185 raft_consensus.cc:3032] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:21.778914 4185 raft_consensus.cc:729] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:11:21.779202 4185 consensus_queue.cc:260] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:21.779727 4185 raft_consensus.cc:3037] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.781572 4447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b" candidate_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "596d3e24651b4787916397f573c09f1d"
I20240501 20:11:21.781833 4447 raft_consensus.cc:3037] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.783272 4185 raft_consensus.cc:2454] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4daecd449e4fc981fd1fe2b0e492c2 in term 2.
I20240501 20:11:21.784868 4447 raft_consensus.cc:2454] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4daecd449e4fc981fd1fe2b0e492c2 in term 2.
I20240501 20:11:21.785445 6081 leader_election.cc:302] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7, ee4daecd449e4fc981fd1fe2b0e492c2; no voters:
I20240501 20:11:21.785759 7246 raft_consensus.cc:2781] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:21.786123 7246 raft_consensus.cc:686] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 LEADER]: Becoming Leader. State: Replica: ee4daecd449e4fc981fd1fe2b0e492c2, State: Running, Role: LEADER
I20240501 20:11:21.786458 7246 consensus_queue.cc:237] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:21.789816 4026 catalog_manager.cc:5484] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: term changed from 1 to 2, leader changed from 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) to ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69). New cstate: current_term: 2 leader_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:21.798069 6281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09b0474ecbf4d458d7a520c9e9f329b" candidate_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f5195730e32a4abd8519baf9961d32f2"
I20240501 20:11:21.798388 6281 raft_consensus.cc:3037] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.798599 4318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da"
dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53798
I20240501 20:11:21.798797 4318 raft_consensus.cc:595] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:21.800868 6281 raft_consensus.cc:2454] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee4daecd449e4fc981fd1fe2b0e492c2 in term 2.
I20240501 20:11:21.832618 9783 ts_tablet_manager.cc:902] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:21.841898 9783 tablet_copy_client.cc:308] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:21.842666 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 618ea2abb259425babf56375f41eb81c from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:53095)
I20240501 20:11:21.842832 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 618ea2abb259425babf56375f41eb81c from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:53095: session id = b662c3da2b1b4f3693920ccb680b8d12-618ea2abb259425babf56375f41eb81c
I20240501 20:11:21.844864 4206 tablet_copy_source_session.cc:205] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:21.845999 9783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ea2abb259425babf56375f41eb81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:21.850342 9783 tablet_copy_client.cc:783] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:21.850569 9783 tablet_copy_client.cc:647] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:21.851960 9787 tablet_copy_client.cc:1002] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:21.873869 4318 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.874505 9517 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.877728 9463 raft_consensus.cc:979] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: : Instructing follower 63c1e379865d42e78ea6b6b1d908ece7 to start an election
I20240501 20:11:21.877902 9463 raft_consensus.cc:1067] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Signalling peer 63c1e379865d42e78ea6b6b1d908ece7 to start an election
I20240501 20:11:21.878687 4185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da"
dest_uuid: "63c1e379865d42e78ea6b6b1d908ece7"
from {username='slave'} at 127.28.136.66:43461
I20240501 20:11:21.878916 4185 raft_consensus.cc:482] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:21.879041 4185 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.881222 4185 raft_consensus.cc:504] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.882057 4185 leader_election.cc:288] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 2 election: Requested vote from peers 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003), b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209)
I20240501 20:11:21.882421 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:21.882503 7466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b662c3da2b1b4f3693920ccb680b8d12"
I20240501 20:11:21.882717 7466 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.882586 6547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "63c1e379865d42e78ea6b6b1d908ece7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "55fe2a4464a9402f927b2b60609b34dd"
I20240501 20:11:21.883028 4318 raft_consensus.cc:3032] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:21.883066 6547 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.883152 4318 raft_consensus.cc:729] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:11:21.883430 4318 consensus_queue.cc:260] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.883893 4318 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:21.884824 7466 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 2.
I20240501 20:11:21.885785 6547 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 2.
I20240501 20:11:21.886302 4116 leader_election.cc:302] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 55fe2a4464a9402f927b2b60609b34dd, 63c1e379865d42e78ea6b6b1d908ece7, b662c3da2b1b4f3693920ccb680b8d12; no voters:
I20240501 20:11:21.886483 4318 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63c1e379865d42e78ea6b6b1d908ece7 in term 2.
I20240501 20:11:21.886628 9531 raft_consensus.cc:2781] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:21.886813 9531 raft_consensus.cc:686] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 LEADER]: Becoming Leader. State: Replica: 63c1e379865d42e78ea6b6b1d908ece7, State: Running, Role: LEADER
I20240501 20:11:21.887104 9531 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:21.890240 4026 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: term changed from 1 to 2, leader changed from 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) to 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65). New cstate: current_term: 2 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:21.930778 7466 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.931360 9517 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:21.935537 6147 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:21.936879 8501 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:22.107715 9827 ts_tablet_manager.cc:902] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:22.108985 9827 tablet_copy_client.cc:308] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:22.109828 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:53095)
I20240501 20:11:22.110056 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:53095: session id = b662c3da2b1b4f3693920ccb680b8d12-3d7b1a47b1fe4e4b9a60b18b0b99056b
I20240501 20:11:22.112232 4206 tablet_copy_source_session.cc:205] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:22.113451 9827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.118188 9827 tablet_copy_client.cc:783] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:22.118413 9827 tablet_copy_client.cc:647] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:22.119720 9831 tablet_copy_client.cc:1002] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:22.220435 4185 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:22.221086 7246 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:22.244344 9833 ts_tablet_manager.cc:902] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
I20240501 20:11:22.246013 9833 tablet_copy_client.cc:308] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38865
I20240501 20:11:22.246654 4469 tablet_copy_service.cc:140] P 596d3e24651b4787916397f573c09f1d: Received BeginTabletCopySession request for tablet c3b4a011c5944a1db9dbc496e87d5768 from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:40053)
I20240501 20:11:22.246809 4469 tablet_copy_service.cc:161] P 596d3e24651b4787916397f573c09f1d: Beginning new tablet copy session on tablet c3b4a011c5944a1db9dbc496e87d5768 from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:40053: session id = b662c3da2b1b4f3693920ccb680b8d12-c3b4a011c5944a1db9dbc496e87d5768
I20240501 20:11:22.248927 4469 tablet_copy_source_session.cc:205] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:22.250130 9833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b4a011c5944a1db9dbc496e87d5768. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.254365 9833 tablet_copy_client.cc:783] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:22.254595 9833 tablet_copy_client.cc:647] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:22.256022 9837 tablet_copy_client.cc:1002] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:22.262296 6281 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:22.262884 7246 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:22.267810 4447 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:22.268430 7246 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:22.304270 7466 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:22.304975 9503 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:22.307719 6547 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:22.308305 9503 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:22.315505 4318 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:22.316123 9503 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:22.318578 4185 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:22.320935 7466 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:22.321110 6546 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:22.321821 9531 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:22.322479 9531 consensus_queue.cc:1035] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:22.324064 9503 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:22.327049 7466 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:22.327519 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eeb1eec290ce4dcbb2afedd57e3b81da with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:22.328272 6547 raft_consensus.cc:2932] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:22.331887 4026 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 2 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:22.337662 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb1eec290ce4dcbb2afedd57e3b81da with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:22.338282 9870 tablet_replica.cc:330] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:22.338493 9870 raft_consensus.cc:2229] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:22.338683 9870 raft_consensus.cc:2258] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:22.340150 9870 ts_tablet_manager.cc:1875] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:22.346671 9870 ts_tablet_manager.cc:1888] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:11:22.346807 9870 log.cc:1196] T eeb1eec290ce4dcbb2afedd57e3b81da P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/eeb1eec290ce4dcbb2afedd57e3b81da
I20240501 20:11:22.347729 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet eeb1eec290ce4dcbb2afedd57e3b81da (table rebalance_test_table_9 [id=236fbeabc92c44f3ae44c8c43af8f5eb]) successfully deleted
I20240501 20:11:22.377339 4318 consensus_queue.cc:237] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:22.380199 6281 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:22.380542 4253 consensus_peers.cc:485] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ff47913ab3b4771abbd9c912bbf34ad. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:22.380529 4577 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:22.380897 9463 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:22.381541 8868 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:22.397045 8868 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:22.398216 6281 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:22.402151 4031 catalog_manager.cc:5484] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:22.404922 4577 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:22.527238 4577 consensus_queue.cc:237] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:22.529038 6147 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:22.529147 7466 raft_consensus.cc:1261] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Refusing update from remote peer 75b5acf03b9f4f6e98a67ae046587e79: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:22.529578 9921 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:22.529891 9922 consensus_queue.cc:1035] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:22.531985 9517 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:22.532547 7466 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:22.534941 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5cdac8c0b92d442ebd84fb43af7e32fb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:22.535465 6147 raft_consensus.cc:2932] T 5cdac8c0b92d442ebd84fb43af7e32fb P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:22.537511 4031 catalog_manager.cc:5484] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 reported cstate change: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 2 leader_uuid: "75b5acf03b9f4f6e98a67ae046587e79" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:22.541791 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdac8c0b92d442ebd84fb43af7e32fb with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:22.542205 9870 tablet_replica.cc:330] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:22.542448 9870 raft_consensus.cc:2229] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:22.542629 9870 raft_consensus.cc:2258] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:22.543366 9870 ts_tablet_manager.cc:1875] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:22.547654 9870 ts_tablet_manager.cc:1888] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:11:22.547761 9870 log.cc:1196] T 5cdac8c0b92d442ebd84fb43af7e32fb P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/5cdac8c0b92d442ebd84fb43af7e32fb
I20240501 20:11:22.548398 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet 5cdac8c0b92d442ebd84fb43af7e32fb (table rebalance_test_table_1 [id=32cdaeaf34cb4443af47a0440ee9a666]) successfully deleted
I20240501 20:11:22.760403 4185 consensus_queue.cc:237] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:22.762984 4318 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:11:22.763027 6416 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:11:22.763203 4118 consensus_peers.cc:485] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 -> Peer b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209): Couldn't send request to peer b662c3da2b1b4f3693920ccb680b8d12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60871b6059c249bfaac713e2f7325811. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:22.763563 9862 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:22.763888 9531 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:11:22.766482 9531 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:22.767204 6416 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:22.773835 4026 catalog_manager.cc:5484] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:22.783468 4318 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } } }
I20240501 20:11:22.854007 6146 consensus_queue.cc:237] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } }
I20240501 20:11:22.855813 6281 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:22.856209 4447 raft_consensus.cc:1261] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Refusing update from remote peer ee4daecd449e4fc981fd1fe2b0e492c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:22.856617 9998 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:22.857458 9997 consensus_queue.cc:1035] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:22.859660 7246 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:22.862917 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d09b0474ecbf4d458d7a520c9e9f329b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:22.863557 4447 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P 596d3e24651b4787916397f573c09f1d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:22.868978 6281 raft_consensus.cc:2932] T d09b0474ecbf4d458d7a520c9e9f329b P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } }
I20240501 20:11:22.869858 4035 catalog_manager.cc:5484] T d09b0474ecbf4d458d7a520c9e9f329b P ee4daecd449e4fc981fd1fe2b0e492c2 reported cstate change: config changed from index 8 to 10, VOTER 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) evicted. New cstate: current_term: 2 leader_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:22.877089 4164 tablet_service.cc:1514] Processing DeleteTablet for tablet d09b0474ecbf4d458d7a520c9e9f329b with delete_type TABLET_DATA_TOMBSTONED (TS 63c1e379865d42e78ea6b6b1d908ece7 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33054
I20240501 20:11:22.884912 10001 tablet_replica.cc:330] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: stopping tablet replica
I20240501 20:11:22.885143 10001 raft_consensus.cc:2229] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:22.885393 10001 raft_consensus.cc:2258] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:22.886385 10001 ts_tablet_manager.cc:1875] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:22.892366 10001 ts_tablet_manager.cc:1888] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:11:22.892539 10001 log.cc:1196] T d09b0474ecbf4d458d7a520c9e9f329b P 63c1e379865d42e78ea6b6b1d908ece7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/d09b0474ecbf4d458d7a520c9e9f329b
I20240501 20:11:22.893378 4018 catalog_manager.cc:4830] TS 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): tablet d09b0474ecbf4d458d7a520c9e9f329b (table rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]) successfully deleted
I20240501 20:11:22.977741 10002 ts_tablet_manager.cc:902] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
I20240501 20:11:22.979053 10002 tablet_copy_client.cc:308] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:38397
I20240501 20:11:22.979769 4337 tablet_copy_service.cc:140] P 4acbc8659ce24119bd77571d1171ee1d: Received BeginTabletCopySession request for tablet 9ff47913ab3b4771abbd9c912bbf34ad from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:45225)
I20240501 20:11:22.979954 4337 tablet_copy_service.cc:161] P 4acbc8659ce24119bd77571d1171ee1d: Beginning new tablet copy session on tablet 9ff47913ab3b4771abbd9c912bbf34ad from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:45225: session id = b662c3da2b1b4f3693920ccb680b8d12-9ff47913ab3b4771abbd9c912bbf34ad
I20240501 20:11:22.982077 4337 tablet_copy_source_session.cc:205] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:22.983264 10002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff47913ab3b4771abbd9c912bbf34ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:22.987990 10002 tablet_copy_client.cc:783] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:22.988204 10002 tablet_copy_client.cc:647] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:22.989698 10007 tablet_copy_client.cc:1002] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:23.195984 10068 ts_tablet_manager.cc:902] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: Initiating tablet copy from peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
I20240501 20:11:23.197474 10068 tablet_copy_client.cc:308] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:43231
I20240501 20:11:23.198225 4206 tablet_copy_service.cc:140] P 63c1e379865d42e78ea6b6b1d908ece7: Received BeginTabletCopySession request for tablet 60871b6059c249bfaac713e2f7325811 from peer b662c3da2b1b4f3693920ccb680b8d12 ({username='slave'} at 127.28.136.73:53095)
I20240501 20:11:23.198390 4206 tablet_copy_service.cc:161] P 63c1e379865d42e78ea6b6b1d908ece7: Beginning new tablet copy session on tablet 60871b6059c249bfaac713e2f7325811 from peer b662c3da2b1b4f3693920ccb680b8d12 at {username='slave'} at 127.28.136.73:53095: session id = b662c3da2b1b4f3693920ccb680b8d12-60871b6059c249bfaac713e2f7325811
I20240501 20:11:23.200480 4206 tablet_copy_source_session.cc:205] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:23.201789 10068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60871b6059c249bfaac713e2f7325811. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:23.206209 10068 tablet_copy_client.cc:783] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:23.206454 10068 tablet_copy_client.cc:647] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:23.208053 10072 tablet_copy_client.cc:1002] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Injecting latency into file download: 1500
I20240501 20:11:23.352321 9783 tablet_copy_client.cc:523] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:23.355523 9783 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:23.402858 9783 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:23.403209 9783 tablet_bootstrap.cc:492] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:23.403458 9783 ts_tablet_manager.cc:1367] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.048s user 0.043s sys 0.002s
I20240501 20:11:23.404078 9783 raft_consensus.cc:348] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:23.404325 9783 raft_consensus.cc:729] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:23.404512 9783 consensus_queue.cc:260] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:23.405184 9783 ts_tablet_manager.cc:1398] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240501 20:11:23.405961 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-618ea2abb259425babf56375f41eb81c received from {username='slave'} at 127.28.136.73:53095
I20240501 20:11:23.406105 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-618ea2abb259425babf56375f41eb81c on tablet 618ea2abb259425babf56375f41eb81c with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:23.496608 7466 raft_consensus.cc:1203] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:23.620107 9827 tablet_copy_client.cc:523] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:23.623418 9827 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:23.661763 9827 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:23.662158 9827 tablet_bootstrap.cc:492] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:23.662423 9827 ts_tablet_manager.cc:1367] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.039s user 0.034s sys 0.004s
I20240501 20:11:23.663053 9827 raft_consensus.cc:348] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:23.663276 9827 raft_consensus.cc:729] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:23.663434 9827 consensus_queue.cc:260] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:23.663939 9827 ts_tablet_manager.cc:1398] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:23.664688 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-3d7b1a47b1fe4e4b9a60b18b0b99056b received from {username='slave'} at 127.28.136.73:53095
I20240501 20:11:23.664860 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-3d7b1a47b1fe4e4b9a60b18b0b99056b on tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:23.756481 9833 tablet_copy_client.cc:523] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:23.759555 9833 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:23.793560 9833 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:23.793915 9833 tablet_bootstrap.cc:492] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:23.794142 9833 ts_tablet_manager.cc:1367] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.035s user 0.033s sys 0.002s
I20240501 20:11:23.794730 9833 raft_consensus.cc:348] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:23.794950 9833 raft_consensus.cc:729] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:23.795099 9833 consensus_queue.cc:260] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:23.795642 9833 ts_tablet_manager.cc:1398] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:23.796522 4469 tablet_copy_service.cc:342] P 596d3e24651b4787916397f573c09f1d: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-c3b4a011c5944a1db9dbc496e87d5768 received from {username='slave'} at 127.28.136.73:40053
I20240501 20:11:23.796656 4469 tablet_copy_service.cc:433] P 596d3e24651b4787916397f573c09f1d: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-c3b4a011c5944a1db9dbc496e87d5768 on tablet c3b4a011c5944a1db9dbc496e87d5768 with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:24.009506 9862 raft_consensus.cc:1050] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:24.010219 9862 consensus_queue.cc:237] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:24.012873 6416 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.012904 7466 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.012936 4577 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.013522 9862 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.013862 9531 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.014148 10162 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.017881 10162 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.018573 6416 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.018918 7466 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.023504 4026 catalog_manager.cc:5484] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.025856 4577 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.031028 4185 consensus_queue.cc:237] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:24.032802 7466 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:24.032914 6416 raft_consensus.cc:1261] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:24.033345 10168 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:24.033871 10168 consensus_queue.cc:1035] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:24.035943 10168 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.037211 6416 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.037241 7466 raft_consensus.cc:2932] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.039813 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 618ea2abb259425babf56375f41eb81c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:24.042222 4031 catalog_manager.cc:5484] T 618ea2abb259425babf56375f41eb81c P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:24.045979 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet 618ea2abb259425babf56375f41eb81c with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:24.046690 10172 tablet_replica.cc:330] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:24.046854 10172 raft_consensus.cc:2229] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:24.047122 10172 raft_consensus.cc:2258] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:24.047842 10172 ts_tablet_manager.cc:1875] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:24.052605 10172 ts_tablet_manager.cc:1888] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:24.052716 10172 log.cc:1196] T 618ea2abb259425babf56375f41eb81c P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/618ea2abb259425babf56375f41eb81c
I20240501 20:11:24.053452 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet 618ea2abb259425babf56375f41eb81c (table rebalance_test_table_4 [id=1c51f8a15c4b42e0bfe8cdfd823d7079]) successfully deleted
I20240501 20:11:24.157501 7466 raft_consensus.cc:1203] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:24.256880 7466 raft_consensus.cc:1203] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:24.490013 10002 tablet_copy_client.cc:523] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:24.492781 10002 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:24.525208 10002 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:24.525712 10002 tablet_bootstrap.cc:492] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:24.526068 10002 ts_tablet_manager.cc:1367] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.033s user 0.023s sys 0.013s
I20240501 20:11:24.526863 10002 raft_consensus.cc:348] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:24.527175 10002 raft_consensus.cc:729] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:24.527437 10002 consensus_queue.cc:260] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:24.528062 10002 ts_tablet_manager.cc:1398] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:24.528965 4337 tablet_copy_service.cc:342] P 4acbc8659ce24119bd77571d1171ee1d: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-9ff47913ab3b4771abbd9c912bbf34ad received from {username='slave'} at 127.28.136.73:45225
I20240501 20:11:24.529114 4337 tablet_copy_service.cc:433] P 4acbc8659ce24119bd77571d1171ee1d: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-9ff47913ab3b4771abbd9c912bbf34ad on tablet 9ff47913ab3b4771abbd9c912bbf34ad with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:24.708433 10068 tablet_copy_client.cc:523] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:24.711195 10068 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap starting.
I20240501 20:11:24.741545 10260 raft_consensus.cc:1050] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:24.742496 10260 consensus_queue.cc:237] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:24.744349 10068 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:24.744861 10068 tablet_bootstrap.cc:492] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: Bootstrap complete.
I20240501 20:11:24.745005 9531 raft_consensus.cc:1050] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:24.745339 10068 ts_tablet_manager.cc:1367] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.010s
I20240501 20:11:24.745940 9531 consensus_queue.cc:237] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:24.746325 10068 raft_consensus.cc:348] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:24.746781 10068 raft_consensus.cc:729] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Initialized, Role: LEARNER
I20240501 20:11:24.747059 10068 consensus_queue.cc:260] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: true } }
I20240501 20:11:24.747743 10068 ts_tablet_manager.cc:1398] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:24.748626 4206 tablet_copy_service.cc:342] P 63c1e379865d42e78ea6b6b1d908ece7: Request end of tablet copy session b662c3da2b1b4f3693920ccb680b8d12-60871b6059c249bfaac713e2f7325811 received from {username='slave'} at 127.28.136.73:53095
I20240501 20:11:24.748809 4206 tablet_copy_service.cc:433] P 63c1e379865d42e78ea6b6b1d908ece7: ending tablet copy session b662c3da2b1b4f3693920ccb680b8d12-60871b6059c249bfaac713e2f7325811 on tablet 60871b6059c249bfaac713e2f7325811 with peer b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:24.751083 7466 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.752233 10267 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.752519 6416 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.753784 4447 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.754787 9531 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.755100 10263 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.762322 9531 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.762949 4447 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.764747 7466 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.768015 6416 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.769022 4031 catalog_manager.cc:5484] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } attrs { replace: true } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.770138 6281 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.770279 7467 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.771044 10261 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.771756 10067 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.772150 4577 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:24.773104 10067 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:24.775760 10261 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.776412 6281 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.782685 4030 catalog_manager.cc:5484] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.784034 4577 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.794685 7466 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } attrs { replace: true } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.800706 4447 consensus_queue.cc:237] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:24.802385 7466 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:24.802490 6281 raft_consensus.cc:1261] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 596d3e24651b4787916397f573c09f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:24.802917 10274 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:24.803193 10067 consensus_queue.cc:1035] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:24.805269 10274 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.805881 6281 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.806141 7466 raft_consensus.cc:2932] T c3b4a011c5944a1db9dbc496e87d5768 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:24.811300 4029 catalog_manager.cc:5484] T c3b4a011c5944a1db9dbc496e87d5768 P 596d3e24651b4787916397f573c09f1d reported cstate change: config changed from index 8 to 9, VOTER 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "596d3e24651b4787916397f573c09f1d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:24.814018 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3b4a011c5944a1db9dbc496e87d5768 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:24.814844 4558 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b4a011c5944a1db9dbc496e87d5768 with delete_type TABLET_DATA_TOMBSTONED (TS 75b5acf03b9f4f6e98a67ae046587e79 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51024
I20240501 20:11:24.815596 10282 tablet_replica.cc:330] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: stopping tablet replica
I20240501 20:11:24.815853 10282 raft_consensus.cc:2229] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:24.816208 10282 raft_consensus.cc:2258] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:24.817346 10282 ts_tablet_manager.cc:1875] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:24.822897 10282 ts_tablet_manager.cc:1888] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:24.823083 10282 log.cc:1196] T c3b4a011c5944a1db9dbc496e87d5768 P 75b5acf03b9f4f6e98a67ae046587e79: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/c3b4a011c5944a1db9dbc496e87d5768
I20240501 20:11:24.823781 4020 catalog_manager.cc:4830] TS 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): tablet c3b4a011c5944a1db9dbc496e87d5768 (table rebalance_test_table_8 [id=1134214a27704fdaa2d3dac65216a93f]) successfully deleted
I20240501 20:11:25.011968 7466 raft_consensus.cc:1203] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:25.057802 7466 raft_consensus.cc:1203] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:11:25.169715 4185 consensus_queue.cc:237] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:25.171535 7466 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:25.171600 6416 raft_consensus.cc:1261] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:25.172070 10268 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:25.172439 10266 consensus_queue.cc:1035] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:25.174782 10263 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.175542 6416 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.175498 7466 raft_consensus.cc:2932] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.179200 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:25.180871 4029 catalog_manager.cc:5484] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 8 to 9, VOTER 596d3e24651b4787916397f573c09f1d (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.187012 4428 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b with delete_type TABLET_DATA_TOMBSTONED (TS 596d3e24651b4787916397f573c09f1d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:32846
I20240501 20:11:25.187768 10324 tablet_replica.cc:330] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: stopping tablet replica
I20240501 20:11:25.188040 10324 raft_consensus.cc:2229] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:25.188664 10324 raft_consensus.cc:2258] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:25.189595 10324 ts_tablet_manager.cc:1875] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:25.196115 10324 ts_tablet_manager.cc:1888] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:25.196256 10324 log.cc:1196] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 596d3e24651b4787916397f573c09f1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d7b1a47b1fe4e4b9a60b18b0b99056b
I20240501 20:11:25.197093 4018 catalog_manager.cc:4830] TS 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b (table rebalance_test_table_0 [id=73592bde195e4b36a460c9befd256e94]) successfully deleted
I20240501 20:11:25.494383 10266 raft_consensus.cc:1050] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:25.495095 10266 consensus_queue.cc:237] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:25.497628 4318 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:25.497622 6416 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:25.498194 7466 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:25.498194 9531 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:25.498530 10267 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:25.498790 10263 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:25.510546 10267 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.511273 6416 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.511972 4318 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.514356 7466 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.515836 4027 catalog_manager.cc:5484] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:25.520542 4185 consensus_queue.cc:237] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:25.521937 10318 raft_consensus.cc:1050] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: attempting to promote NON_VOTER b662c3da2b1b4f3693920ccb680b8d12 to VOTER
I20240501 20:11:25.522720 7466 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:25.522811 6416 raft_consensus.cc:1261] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Refusing update from remote peer 63c1e379865d42e78ea6b6b1d908ece7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:11:25.522764 10318 consensus_queue.cc:237] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:25.523334 10267 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:25.525748 7466 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 1 LEARNER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:25.525923 6281 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:25.526383 10108 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:25.526705 9531 consensus_queue.cc:1035] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:25.526958 10108 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:25.527407 10263 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 63c1e379865d42e78ea6b6b1d908ece7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.527917 6416 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.532722 4018 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60871b6059c249bfaac713e2f7325811 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:25.535874 4027 catalog_manager.cc:5484] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 reported cstate change: config changed from index 8 to 9, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "63c1e379865d42e78ea6b6b1d908ece7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.537349 7466 raft_consensus.cc:2932] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.543205 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet 60871b6059c249bfaac713e2f7325811 with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:25.547396 10364 tablet_replica.cc:330] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:25.547653 10364 raft_consensus.cc:2229] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:25.549022 10364 raft_consensus.cc:2258] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:25.549435 10108 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.550392 10364 ts_tablet_manager.cc:1875] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:25.553607 4577 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Refusing update from remote peer 4acbc8659ce24119bd77571d1171ee1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:11:25.553490 7467 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.554315 10108 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:11:25.555980 4027 catalog_manager.cc:5484] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d reported cstate change: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4acbc8659ce24119bd77571d1171ee1d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:25.557679 10364 ts_tablet_manager.cc:1888] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:11:25.557847 10364 log.cc:1196] T 60871b6059c249bfaac713e2f7325811 P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/60871b6059c249bfaac713e2f7325811
I20240501 20:11:25.557526 6281 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.558687 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet 60871b6059c249bfaac713e2f7325811 (table rebalance_test_table_7 [id=7a4db4b8c212476d8b3920fa54833783]) successfully deleted
I20240501 20:11:25.561123 4578 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:25.687726 4318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad"
dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53974
I20240501 20:11:25.687922 4318 raft_consensus.cc:595] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:25.705590 4318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad"
dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53996
I20240501 20:11:25.705839 4318 raft_consensus.cc:595] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:25.928128 4318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad"
dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54012
I20240501 20:11:25.928314 4318 raft_consensus.cc:595] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Received request to transfer leadership
I20240501 20:11:25.986866 10365 raft_consensus.cc:979] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: : Instructing follower b662c3da2b1b4f3693920ccb680b8d12 to start an election
I20240501 20:11:25.987088 10318 raft_consensus.cc:1067] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Signalling peer b662c3da2b1b4f3693920ccb680b8d12 to start an election
I20240501 20:11:25.987725 7467 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad"
dest_uuid: "b662c3da2b1b4f3693920ccb680b8d12"
from {username='slave'} at 127.28.136.66:37791
I20240501 20:11:25.988051 7467 raft_consensus.cc:482] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:25.988178 7467 raft_consensus.cc:3037] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:25.990298 7467 raft_consensus.cc:504] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:25.991919 7467 leader_election.cc:288] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 election: Requested vote from peers 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397), 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125), f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983)
I20240501 20:11:25.992230 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad" candidate_uuid: "b662c3da2b1b4f3693920ccb680b8d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4acbc8659ce24119bd77571d1171ee1d"
I20240501 20:11:25.992683 4318 raft_consensus.cc:3032] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:25.992811 4318 raft_consensus.cc:729] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4acbc8659ce24119bd77571d1171ee1d, State: Running, Role: LEADER
I20240501 20:11:25.993074 4318 consensus_queue.cc:260] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:25.993512 4318 raft_consensus.cc:3037] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:25.995944 4318 raft_consensus.cc:2454] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b662c3da2b1b4f3693920ccb680b8d12 in term 2.
I20240501 20:11:25.996601 6281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad" candidate_uuid: "b662c3da2b1b4f3693920ccb680b8d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f5195730e32a4abd8519baf9961d32f2"
I20240501 20:11:25.996855 6281 raft_consensus.cc:3037] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:25.999368 6281 raft_consensus.cc:2454] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b662c3da2b1b4f3693920ccb680b8d12 in term 2.
I20240501 20:11:25.999907 4578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff47913ab3b4771abbd9c912bbf34ad" candidate_uuid: "b662c3da2b1b4f3693920ccb680b8d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "75b5acf03b9f4f6e98a67ae046587e79"
I20240501 20:11:25.999909 7336 leader_election.cc:302] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4acbc8659ce24119bd77571d1171ee1d, b662c3da2b1b4f3693920ccb680b8d12, f5195730e32a4abd8519baf9961d32f2; no voters:
I20240501 20:11:26.000125 4578 raft_consensus.cc:3037] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:26.000312 10106 raft_consensus.cc:2781] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:26.001068 10106 raft_consensus.cc:686] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 2 LEADER]: Becoming Leader. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Running, Role: LEADER
I20240501 20:11:26.001585 10106 consensus_queue.cc:237] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:26.002838 4578 raft_consensus.cc:2454] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b662c3da2b1b4f3693920ccb680b8d12 in term 2.
I20240501 20:11:26.005926 4027 catalog_manager.cc:5484] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 reported cstate change: term changed from 1 to 2, leader changed from 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) to b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73). New cstate: current_term: 2 leader_uuid: "b662c3da2b1b4f3693920ccb680b8d12" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:26.405525 6281 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Refusing update from remote peer b662c3da2b1b4f3693920ccb680b8d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:26.406208 10106 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:26.413884 4578 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Refusing update from remote peer b662c3da2b1b4f3693920ccb680b8d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:26.414402 10106 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:26.418174 4318 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Refusing update from remote peer b662c3da2b1b4f3693920ccb680b8d12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:11:26.418905 10106 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:11:26.468299 7467 consensus_queue.cc:237] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:26.470317 6281 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Refusing update from remote peer b662c3da2b1b4f3693920ccb680b8d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:26.470803 4578 raft_consensus.cc:1261] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Refusing update from remote peer b662c3da2b1b4f3693920ccb680b8d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:11:26.470896 10106 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:26.471617 10106 consensus_queue.cc:1035] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:11:26.473104 10418 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:26.473861 6281 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P f5195730e32a4abd8519baf9961d32f2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:26.474411 4578 raft_consensus.cc:2932] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:26.479703 4020 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ff47913ab3b4771abbd9c912bbf34ad with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:11:26.479772 4030 catalog_manager.cc:5484] T 9ff47913ab3b4771abbd9c912bbf34ad P 75b5acf03b9f4f6e98a67ae046587e79 reported cstate change: config changed from index 8 to 10, VOTER 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66) evicted. New cstate: current_term: 2 leader_uuid: "b662c3da2b1b4f3693920ccb680b8d12" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } } }
I20240501 20:11:26.483098 4296 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff47913ab3b4771abbd9c912bbf34ad with delete_type TABLET_DATA_TOMBSTONED (TS 4acbc8659ce24119bd77571d1171ee1d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35312
I20240501 20:11:26.483811 10424 tablet_replica.cc:330] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: stopping tablet replica
I20240501 20:11:26.483985 10424 raft_consensus.cc:2229] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:11:26.484172 10424 raft_consensus.cc:2258] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:11:26.485045 10424 ts_tablet_manager.cc:1875] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:11:26.489884 10424 ts_tablet_manager.cc:1888] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:11:26.490003 10424 log.cc:1196] T 9ff47913ab3b4771abbd9c912bbf34ad P 4acbc8659ce24119bd77571d1171ee1d: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/9ff47913ab3b4771abbd9c912bbf34ad
I20240501 20:11:26.490808 4020 catalog_manager.cc:4830] TS 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): tablet 9ff47913ab3b4771abbd9c912bbf34ad (table rebalance_test_table_5 [id=621c0cb0d692492d9486b2cab5397031]) successfully deleted
I20240501 20:11:26.693989 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:26.722514 4428 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:26.728459 4296 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:26.731256 4164 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:26.733026 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:26.741019 7420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:26.749900 4558 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:26.767398 6261 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:26.796696 6128 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:27.023797 4296 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:27.026679 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:27.056378 7420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:27.059188 4558 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:27.067353 4428 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:27.070286 4164 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:27.071417 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:27.103904 6261 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:27.106889 6128 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:28.081243 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.114501 4164 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:28.114645 4428 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:28.180227 4296 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:28.205031 4558 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:28.207230 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.257560 7420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:28.285665 6128 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:28.307857 6261 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:28.527304 4296 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:28.541150 4428 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:28.543754 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.568707 4164 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:28.586341 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.590802 4558 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:28.618144 6128 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:28.624863 7420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:28.639222 6261 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:28.841696 4296 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:28.867960 4428 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:28.876849 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.880903 4558 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:28.886142 4164 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:28.903662 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:28.924198 7420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:28.948835 6128 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:28.964874 6261 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:29.300634 6527 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:29.302031 4296 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:29.312482 4558 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:29.338802 4428 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:11:29.339474 4164 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:29.356937 7420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:29.399320 6396 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:29.409456 6128 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:11:29.432368 6261 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7618d84946ab4e0096d32aaa40ff607d | 127.28.136.126:33949 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.28.136.84:34555 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
4acbc8659ce24119bd77571d1171ee1d | 127.28.136.66:38397 | HEALTHY | <none> | 7 | 0
55fe2a4464a9402f927b2b60609b34dd | 127.28.136.72:46003 | HEALTHY | <none> | 0 | 0
596d3e24651b4787916397f573c09f1d | 127.28.136.67:38865 | HEALTHY | <none> | 4 | 0
63c1e379865d42e78ea6b6b1d908ece7 | 127.28.136.65:43231 | HEALTHY | <none> | 8 | 0
75b5acf03b9f4f6e98a67ae046587e79 | 127.28.136.68:36125 | HEALTHY | <none> | 7 | 0
9592068f16ca406eb8a347a65d365782 | 127.28.136.71:40971 | HEALTHY | <none> | 0 | 0
b662c3da2b1b4f3693920ccb680b8d12 | 127.28.136.73:46209 | HEALTHY | <none> | 1 | 0
ee4daecd449e4fc981fd1fe2b0e492c2 | 127.28.136.69:45155 | HEALTHY | <none> | 2 | 0
f5195730e32a4abd8519baf9961d32f2 | 127.28.136.70:40983 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.28.136.65 | experimental | 127.28.136.65:43231
local_ip_for_outbound_sockets | 127.28.136.66 | experimental | 127.28.136.66:38397
local_ip_for_outbound_sockets | 127.28.136.67 | experimental | 127.28.136.67:38865
local_ip_for_outbound_sockets | 127.28.136.68 | experimental | 127.28.136.68:36125
local_ip_for_outbound_sockets | 127.28.136.69 | experimental | 127.28.136.69:45155
local_ip_for_outbound_sockets | 127.28.136.70 | experimental | 127.28.136.70:40983
local_ip_for_outbound_sockets | 127.28.136.71 | experimental | 127.28.136.71:40971
local_ip_for_outbound_sockets | 127.28.136.72 | experimental | 127.28.136.72:46003
local_ip_for_outbound_sockets | 127.28.136.73 | experimental | 127.28.136.73:46209
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.136.65:43231
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.136.66:38397
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.136.67:38865
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.28.136.68:36125
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.28.136.69:45155
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.28.136.70:40983
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.28.136.71:40971
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.28.136.72:46003
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.28.136.73:46209
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.28.136.84:34555 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:11:29.523275 29217 log_verifier.cc:126] Checking tablet 0822a3fc82134d018bae96fecc62fdd3
I20240501 20:11:29.563303 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 0822a3fc82134d018bae96fecc62fdd3
I20240501 20:11:29.563416 29217 log_verifier.cc:126] Checking tablet 0a452749c5c44b87bb495983853cea3a
I20240501 20:11:29.601452 29217 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 0a452749c5c44b87bb495983853cea3a
I20240501 20:11:29.601552 29217 log_verifier.cc:126] Checking tablet 11f240637b1e4ef8bc3414e11b38f469
I20240501 20:11:29.640635 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 11f240637b1e4ef8bc3414e11b38f469
I20240501 20:11:29.640790 29217 log_verifier.cc:126] Checking tablet 126f7b0e88774d51ac087e5a1f758c78
I20240501 20:11:29.679370 29217 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 126f7b0e88774d51ac087e5a1f758c78
I20240501 20:11:29.679483 29217 log_verifier.cc:126] Checking tablet 13d3d6724e3242478fef21f83dc94452
I20240501 20:11:29.754794 29217 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 13d3d6724e3242478fef21f83dc94452
I20240501 20:11:29.754882 29217 log_verifier.cc:126] Checking tablet 1564f07420a346dbaa77ef6a2a1a1b42
I20240501 20:11:29.791987 29217 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1564f07420a346dbaa77ef6a2a1a1b42
I20240501 20:11:29.792084 29217 log_verifier.cc:126] Checking tablet 28b9b57213a64172afdc2ca890569fc5
I20240501 20:11:29.828933 29217 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 28b9b57213a64172afdc2ca890569fc5
I20240501 20:11:29.829030 29217 log_verifier.cc:126] Checking tablet 2e97fe7ab97a4458b5e977af3d1356df
I20240501 20:11:29.874102 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 2e97fe7ab97a4458b5e977af3d1356df
I20240501 20:11:29.874228 29217 log_verifier.cc:126] Checking tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b
I20240501 20:11:29.915616 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 3d7b1a47b1fe4e4b9a60b18b0b99056b
I20240501 20:11:29.915748 29217 log_verifier.cc:126] Checking tablet 3f7fb3f245414d05a8e5d837076195c2
I20240501 20:11:29.961292 29217 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 3f7fb3f245414d05a8e5d837076195c2
I20240501 20:11:29.961407 29217 log_verifier.cc:126] Checking tablet 5cdac8c0b92d442ebd84fb43af7e32fb
I20240501 20:11:30.037027 29217 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 5cdac8c0b92d442ebd84fb43af7e32fb
I20240501 20:11:30.037137 29217 log_verifier.cc:126] Checking tablet 60871b6059c249bfaac713e2f7325811
I20240501 20:11:30.075363 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 60871b6059c249bfaac713e2f7325811
I20240501 20:11:30.075495 29217 log_verifier.cc:126] Checking tablet 618ea2abb259425babf56375f41eb81c
I20240501 20:11:30.112671 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 618ea2abb259425babf56375f41eb81c
I20240501 20:11:30.112829 29217 log_verifier.cc:126] Checking tablet 79cb3f8b3fe94497bb54d3d8b7f18754
I20240501 20:11:30.149292 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 79cb3f8b3fe94497bb54d3d8b7f18754
I20240501 20:11:30.149412 29217 log_verifier.cc:126] Checking tablet 7d320136fd1d4eb895362144ef6a5322
I20240501 20:11:30.187326 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7d320136fd1d4eb895362144ef6a5322
I20240501 20:11:30.187489 29217 log_verifier.cc:126] Checking tablet 96005b2bfa9847bb992640ad40a68b36
I20240501 20:11:30.227229 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 96005b2bfa9847bb992640ad40a68b36
I20240501 20:11:30.227342 29217 log_verifier.cc:126] Checking tablet 9ff47913ab3b4771abbd9c912bbf34ad
I20240501 20:11:30.301456 29217 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 9ff47913ab3b4771abbd9c912bbf34ad
I20240501 20:11:30.301561 29217 log_verifier.cc:126] Checking tablet ae0fe7a1daab476391d80406766b3b11
I20240501 20:11:30.338678 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet ae0fe7a1daab476391d80406766b3b11
I20240501 20:11:30.338786 29217 log_verifier.cc:126] Checking tablet b344760b82cb4bc09379d990f9558121
I20240501 20:11:30.374320 29217 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b344760b82cb4bc09379d990f9558121
I20240501 20:11:30.374433 29217 log_verifier.cc:126] Checking tablet bb36776ab8b74586aead4ecad56cc815
I20240501 20:11:30.413547 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet bb36776ab8b74586aead4ecad56cc815
I20240501 20:11:30.413674 29217 log_verifier.cc:126] Checking tablet bbd253f3e7824a81b6fdaf2c8880aaa9
I20240501 20:11:30.457401 29217 log_verifier.cc:177] Verified matching terms for 7 ops in tablet bbd253f3e7824a81b6fdaf2c8880aaa9
I20240501 20:11:30.457592 29217 log_verifier.cc:126] Checking tablet bbf736b8dba9474da5fb7ca9d9348ac7
I20240501 20:11:30.533177 29217 log_verifier.cc:177] Verified matching terms for 6 ops in tablet bbf736b8dba9474da5fb7ca9d9348ac7
I20240501 20:11:30.533269 29217 log_verifier.cc:126] Checking tablet c11109bc41e34ab58fe3c9f568adf85e
I20240501 20:11:30.571236 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet c11109bc41e34ab58fe3c9f568adf85e
I20240501 20:11:30.571336 29217 log_verifier.cc:126] Checking tablet c3b4a011c5944a1db9dbc496e87d5768
I20240501 20:11:30.608927 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet c3b4a011c5944a1db9dbc496e87d5768
I20240501 20:11:30.609043 29217 log_verifier.cc:126] Checking tablet ca4d34ed427e44ed88379eb69feaeb8a
I20240501 20:11:30.647779 29217 log_verifier.cc:177] Verified matching terms for 10 ops in tablet ca4d34ed427e44ed88379eb69feaeb8a
I20240501 20:11:30.647884 29217 log_verifier.cc:126] Checking tablet d09b0474ecbf4d458d7a520c9e9f329b
I20240501 20:11:30.685534 29217 log_verifier.cc:177] Verified matching terms for 10 ops in tablet d09b0474ecbf4d458d7a520c9e9f329b
I20240501 20:11:30.685667 29217 log_verifier.cc:126] Checking tablet d1e01586b0ba42869481577ceacc84e8
I20240501 20:11:30.721822 29217 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d1e01586b0ba42869481577ceacc84e8
I20240501 20:11:30.721997 29217 log_verifier.cc:126] Checking tablet d48a2b58ce2d417e9abf63c292419d14
I20240501 20:11:30.799180 29217 log_verifier.cc:177] Verified matching terms for 9 ops in tablet d48a2b58ce2d417e9abf63c292419d14
I20240501 20:11:30.799273 29217 log_verifier.cc:126] Checking tablet e6dc1dbd00224332941bc3301377eb87
I20240501 20:11:30.837788 29217 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e6dc1dbd00224332941bc3301377eb87
I20240501 20:11:30.837888 29217 log_verifier.cc:126] Checking tablet eeb1eec290ce4dcbb2afedd57e3b81da
I20240501 20:11:30.881837 29217 log_verifier.cc:177] Verified matching terms for 10 ops in tablet eeb1eec290ce4dcbb2afedd57e3b81da
I20240501 20:11:30.882648 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 4098
I20240501 20:11:30.889343 4224 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:31.132936 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 4098
I20240501 20:11:31.169322 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 4235
W20240501 20:11:31.169510 4382 connection.cc:535] server connection from 127.28.136.65:44775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:31.169826 4513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.136.65:43231 (error 108)
W20240501 20:11:31.172111 4513 consensus_peers.cc:485] T bbf736b8dba9474da5fb7ca9d9348ac7 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Couldn't send request to peer 63c1e379865d42e78ea6b6b1d908ece7. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:31.176246 4356 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:11:31.410354 6081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
W20240501 20:11:31.411816 6081 consensus_peers.cc:485] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 -> Peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Couldn't send request to peer 63c1e379865d42e78ea6b6b1d908ece7. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:31.441416 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 4235
W20240501 20:11:31.487888 6351 connection.cc:535] server connection from 127.28.136.66:38739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:31.488113 4515 connection.cc:535] server connection from 127.28.136.66:54739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:11:31.488382 4381 connection.cc:535] server connection from 127.28.136.66:42967 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:31.488598 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 4366
W20240501 20:11:31.489558 4515 consensus_peers.cc:485] T d1e01586b0ba42869481577ceacc84e8 P 75b5acf03b9f4f6e98a67ae046587e79 -> Peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): Couldn't send request to peer 4acbc8659ce24119bd77571d1171ee1d. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.66:38397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:31.495041 4487 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:31.704998 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 4366
I20240501 20:11:31.736752 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 4497
I20240501 20:11:31.742782 4618 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:11:31.770876 6217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.66:38397: connect: Connection refused (error 111)
W20240501 20:11:31.772307 6217 consensus_peers.cc:485] T 3f7fb3f245414d05a8e5d837076195c2 P f5195730e32a4abd8519baf9961d32f2 -> Peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): Couldn't send request to peer 4acbc8659ce24119bd77571d1171ee1d. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.66:38397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:31.780689 6081 consensus_peers.cc:485] T bbd253f3e7824a81b6fdaf2c8880aaa9 P ee4daecd449e4fc981fd1fe2b0e492c2 -> Peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): Couldn't send request to peer 596d3e24651b4787916397f573c09f1d. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.67:38865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:31.879171 10735 raft_consensus.cc:482] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:31.879438 10735 raft_consensus.cc:504] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:31.881235 10735 leader_election.cc:288] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209)
W20240501 20:11:31.894363 6481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
W20240501 20:11:31.896101 6481 leader_election.cc:334] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:31.900774 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "55fe2a4464a9402f927b2b60609b34dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "b662c3da2b1b4f3693920ccb680b8d12" is_pre_election: true
I20240501 20:11:31.901469 6483 leader_election.cc:302] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55fe2a4464a9402f927b2b60609b34dd; no voters: 63c1e379865d42e78ea6b6b1d908ece7, b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:31.901839 10735 raft_consensus.cc:2726] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:11:31.958468 10735 raft_consensus.cc:482] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:31.958709 10735 raft_consensus.cc:504] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:31.959571 10735 leader_election.cc:288] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231)
W20240501 20:11:31.961107 6481 leader_election.cc:334] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): Network error: Client connection negotiation failed: client connection to 127.28.136.67:38865: connect: Connection refused (error 111)
W20240501 20:11:31.961304 6481 leader_election.cc:334] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:31.961421 6481 leader_election.cc:302] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55fe2a4464a9402f927b2b60609b34dd; no voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7
I20240501 20:11:31.961704 10735 raft_consensus.cc:2726] T 0a452749c5c44b87bb495983853cea3a P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:31.972191 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 4497
W20240501 20:11:32.004545 6483 connection.cc:535] server connection from 127.28.136.68:56931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:32.005322 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 6065
I20240501 20:11:32.011718 6187 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:32.106927 10519 raft_consensus.cc:482] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.107230 10519 raft_consensus.cc:504] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.108425 10519 leader_election.cc:288] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971)
W20240501 20:11:32.110451 7335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
W20240501 20:11:32.112188 7335 leader_election.cc:334] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.116154 6415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ea2abb259425babf56375f41eb81c" candidate_uuid: "b662c3da2b1b4f3693920ccb680b8d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9592068f16ca406eb8a347a65d365782" is_pre_election: true
I20240501 20:11:32.116927 7335 leader_election.cc:302] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b662c3da2b1b4f3693920ccb680b8d12; no voters: 63c1e379865d42e78ea6b6b1d908ece7, 9592068f16ca406eb8a347a65d365782
I20240501 20:11:32.117214 10519 raft_consensus.cc:2726] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:32.128515 10735 raft_consensus.cc:482] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.128783 10735 raft_consensus.cc:504] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:32.129398 10735 leader_election.cc:288] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865)
W20240501 20:11:32.131209 6481 leader_election.cc:334] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
W20240501 20:11:32.131392 6481 leader_election.cc:334] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): Network error: Client connection negotiation failed: client connection to 127.28.136.67:38865: connect: Connection refused (error 111)
I20240501 20:11:32.131492 6481 leader_election.cc:302] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55fe2a4464a9402f927b2b60609b34dd; no voters: 596d3e24651b4787916397f573c09f1d, 63c1e379865d42e78ea6b6b1d908ece7
I20240501 20:11:32.131726 10735 raft_consensus.cc:2726] T 1564f07420a346dbaa77ef6a2a1a1b42 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:11:32.132141 7337 consensus_peers.cc:485] T 9ff47913ab3b4771abbd9c912bbf34ad P b662c3da2b1b4f3693920ccb680b8d12 -> Peer 75b5acf03b9f4f6e98a67ae046587e79 (127.28.136.68:36125): Couldn't send request to peer 75b5acf03b9f4f6e98a67ae046587e79. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.68:36125: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.166702 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 6065
I20240501 20:11:32.170779 10519 raft_consensus.cc:482] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.170948 10519 raft_consensus.cc:504] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.171727 10519 leader_election.cc:288] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003)
W20240501 20:11:32.173023 7335 leader_election.cc:334] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.178257 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "b662c3da2b1b4f3693920ccb680b8d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "55fe2a4464a9402f927b2b60609b34dd" is_pre_election: true
I20240501 20:11:32.178512 6546 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b662c3da2b1b4f3693920ccb680b8d12 in term 2.
I20240501 20:11:32.178928 7336 leader_election.cc:302] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55fe2a4464a9402f927b2b60609b34dd, b662c3da2b1b4f3693920ccb680b8d12; no voters: 63c1e379865d42e78ea6b6b1d908ece7
I20240501 20:11:32.179214 10519 raft_consensus.cc:2781] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:11:32.179338 10519 raft_consensus.cc:482] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.179433 10519 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:32.182258 10519 raft_consensus.cc:504] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.183001 10519 leader_election.cc:288] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 3 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003)
I20240501 20:11:32.183516 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeb1eec290ce4dcbb2afedd57e3b81da" candidate_uuid: "b662c3da2b1b4f3693920ccb680b8d12" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "55fe2a4464a9402f927b2b60609b34dd"
I20240501 20:11:32.183713 6546 raft_consensus.cc:3037] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:32.185570 6546 raft_consensus.cc:2454] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b662c3da2b1b4f3693920ccb680b8d12 in term 3.
I20240501 20:11:32.185976 7336 leader_election.cc:302] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fe2a4464a9402f927b2b60609b34dd, b662c3da2b1b4f3693920ccb680b8d12; no voters:
I20240501 20:11:32.186266 10519 raft_consensus.cc:2781] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 3 FOLLOWER]: Leader election won for term 3
W20240501 20:11:32.186743 7335 leader_election.cc:334] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.187125 10519 raft_consensus.cc:686] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [term 3 LEADER]: Becoming Leader. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Running, Role: LEADER
I20240501 20:11:32.187500 10519 consensus_queue.cc:237] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.191098 4032 catalog_manager.cc:5484] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 reported cstate change: term changed from 2 to 3, leader changed from 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) to b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73). New cstate: current_term: 3 leader_uuid: "b662c3da2b1b4f3693920ccb680b8d12" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:32.194813 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 6197
I20240501 20:11:32.201231 6322 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:32.208554 10761 raft_consensus.cc:482] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.208833 10761 raft_consensus.cc:504] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.210355 10761 leader_election.cc:288] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209)
W20240501 20:11:32.211622 6350 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
W20240501 20:11:32.213444 6350 leader_election.cc:334] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.217000 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "9592068f16ca406eb8a347a65d365782" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b662c3da2b1b4f3693920ccb680b8d12" is_pre_election: true
I20240501 20:11:32.217620 6352 leader_election.cc:302] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9592068f16ca406eb8a347a65d365782; no voters: 63c1e379865d42e78ea6b6b1d908ece7, b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:32.217983 10761 raft_consensus.cc:2726] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:32.226399 10761 raft_consensus.cc:482] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.226619 10761 raft_consensus.cc:504] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.227406 10761 leader_election.cc:288] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209)
I20240501 20:11:32.227922 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ea2abb259425babf56375f41eb81c" candidate_uuid: "9592068f16ca406eb8a347a65d365782" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b662c3da2b1b4f3693920ccb680b8d12" is_pre_election: true
I20240501 20:11:32.228197 7467 raft_consensus.cc:2454] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9592068f16ca406eb8a347a65d365782 in term 1.
I20240501 20:11:32.228662 6352 leader_election.cc:302] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9592068f16ca406eb8a347a65d365782, b662c3da2b1b4f3693920ccb680b8d12; no voters:
I20240501 20:11:32.229004 10761 raft_consensus.cc:2781] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 20:11:32.229004 6350 leader_election.cc:334] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.229142 10761 raft_consensus.cc:482] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.229259 10761 raft_consensus.cc:3037] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:32.231277 10761 raft_consensus.cc:504] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.231889 10761 leader_election.cc:288] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209)
I20240501 20:11:32.232448 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ea2abb259425babf56375f41eb81c" candidate_uuid: "9592068f16ca406eb8a347a65d365782" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b662c3da2b1b4f3693920ccb680b8d12"
I20240501 20:11:32.232656 7467 raft_consensus.cc:3037] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:32.234539 7467 raft_consensus.cc:2454] T 618ea2abb259425babf56375f41eb81c P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9592068f16ca406eb8a347a65d365782 in term 2.
I20240501 20:11:32.234987 6352 leader_election.cc:302] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9592068f16ca406eb8a347a65d365782, b662c3da2b1b4f3693920ccb680b8d12; no voters:
I20240501 20:11:32.235271 10761 raft_consensus.cc:2781] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 20:11:32.235249 6350 leader_election.cc:334] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.235797 10761 raft_consensus.cc:686] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [term 2 LEADER]: Becoming Leader. State: Replica: 9592068f16ca406eb8a347a65d365782, State: Running, Role: LEADER
I20240501 20:11:32.236297 10761 consensus_queue.cc:237] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.240214 4032 catalog_manager.cc:5484] T 618ea2abb259425babf56375f41eb81c P 9592068f16ca406eb8a347a65d365782 reported cstate change: term changed from 1 to 2, leader changed from 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) to 9592068f16ca406eb8a347a65d365782 (127.28.136.71). New cstate: current_term: 2 leader_uuid: "9592068f16ca406eb8a347a65d365782" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:32.307796 10761 raft_consensus.cc:482] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.308037 10761 raft_consensus.cc:504] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } }
I20240501 20:11:32.308780 10761 leader_election.cc:288] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155)
W20240501 20:11:32.310274 6350 leader_election.cc:334] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
W20240501 20:11:32.310488 6350 leader_election.cc:334] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee4daecd449e4fc981fd1fe2b0e492c2 (127.28.136.69:45155): Network error: Client connection negotiation failed: client connection to 127.28.136.69:45155: connect: Connection refused (error 111)
I20240501 20:11:32.310611 6350 leader_election.cc:302] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9592068f16ca406eb8a347a65d365782; no voters: 63c1e379865d42e78ea6b6b1d908ece7, ee4daecd449e4fc981fd1fe2b0e492c2
I20240501 20:11:32.310881 10761 raft_consensus.cc:2726] T 96005b2bfa9847bb992640ad40a68b36 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:32.337558 10519 raft_consensus.cc:482] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.337813 10519 raft_consensus.cc:504] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.338493 10519 leader_election.cc:288] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971)
I20240501 20:11:32.339541 6415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "b662c3da2b1b4f3693920ccb680b8d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9592068f16ca406eb8a347a65d365782" is_pre_election: true
I20240501 20:11:32.339852 6415 raft_consensus.cc:2454] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b662c3da2b1b4f3693920ccb680b8d12 in term 1.
W20240501 20:11:32.339951 7335 leader_election.cc:334] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.340277 7335 leader_election.cc:302] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9592068f16ca406eb8a347a65d365782, b662c3da2b1b4f3693920ccb680b8d12; no voters: 63c1e379865d42e78ea6b6b1d908ece7
I20240501 20:11:32.340554 10519 raft_consensus.cc:2781] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:32.340660 10519 raft_consensus.cc:482] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.340796 10519 raft_consensus.cc:3037] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:32.342728 10519 raft_consensus.cc:504] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.343266 10519 leader_election.cc:288] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 election: Requested vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971)
I20240501 20:11:32.344002 6415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d7b1a47b1fe4e4b9a60b18b0b99056b" candidate_uuid: "b662c3da2b1b4f3693920ccb680b8d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9592068f16ca406eb8a347a65d365782"
I20240501 20:11:32.344219 6415 raft_consensus.cc:3037] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:32.344512 7335 leader_election.cc:334] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.346326 6415 raft_consensus.cc:2454] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P 9592068f16ca406eb8a347a65d365782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b662c3da2b1b4f3693920ccb680b8d12 in term 2.
I20240501 20:11:32.346745 7335 leader_election.cc:302] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9592068f16ca406eb8a347a65d365782, b662c3da2b1b4f3693920ccb680b8d12; no voters: 63c1e379865d42e78ea6b6b1d908ece7
I20240501 20:11:32.346983 10519 raft_consensus.cc:2781] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:32.347191 10519 raft_consensus.cc:686] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [term 2 LEADER]: Becoming Leader. State: Replica: b662c3da2b1b4f3693920ccb680b8d12, State: Running, Role: LEADER
I20240501 20:11:32.347502 10519 consensus_queue.cc:237] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.350632 4032 catalog_manager.cc:5484] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 reported cstate change: term changed from 1 to 2, leader changed from 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65) to b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73). New cstate: current_term: 2 leader_uuid: "b662c3da2b1b4f3693920ccb680b8d12" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:11:32.359112 10732 raft_consensus.cc:482] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.359302 10732 raft_consensus.cc:504] T 7d320136fd1d4eb895362144ef6a5322 P f5195730e32a4abd8519baf9961d32f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.359431 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 6197
I20240501 20:11:32.364176 10519 raft_consensus.cc:482] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.364341 10519 raft_consensus.cc:504] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "f5195730e32a4abd8519baf9961d32f2" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 40983 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.365001 10519 leader_election.cc:288] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983)
W20240501 20:11:32.366185 7335 leader_election.cc:334] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.371027 10761 raft_consensus.cc:482] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.371273 10761 raft_consensus.cc:504] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.372040 10761 leader_election.cc:288] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), b662c3da2b1b4f3693920ccb680b8d12 (127.28.136.73:46209)
I20240501 20:11:32.372609 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60871b6059c249bfaac713e2f7325811" candidate_uuid: "9592068f16ca406eb8a347a65d365782" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b662c3da2b1b4f3693920ccb680b8d12" is_pre_election: true
W20240501 20:11:32.373452 6350 leader_election.cc:334] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
I20240501 20:11:32.373585 6350 leader_election.cc:302] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9592068f16ca406eb8a347a65d365782; no voters: 63c1e379865d42e78ea6b6b1d908ece7, b662c3da2b1b4f3693920ccb680b8d12
I20240501 20:11:32.373837 10761 raft_consensus.cc:2726] T 60871b6059c249bfaac713e2f7325811 P 9592068f16ca406eb8a347a65d365782 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:11:32.383744 7336 connection.cc:535] client connection to 127.28.136.70:40983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:11:32.384320 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 6332
W20240501 20:11:32.384963 7336 leader_election.cc:334] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5195730e32a4abd8519baf9961d32f2 (127.28.136.70:40983): Network error: Client connection negotiation failed: client connection to 127.28.136.70:40983: connect: Connection refused (error 111)
I20240501 20:11:32.385102 7336 leader_election.cc:302] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b662c3da2b1b4f3693920ccb680b8d12; no voters: 63c1e379865d42e78ea6b6b1d908ece7, f5195730e32a4abd8519baf9961d32f2
I20240501 20:11:32.385406 10519 raft_consensus.cc:2726] T 7d320136fd1d4eb895362144ef6a5322 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:32.389997 6457 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:32.456089 10735 raft_consensus.cc:482] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4acbc8659ce24119bd77571d1171ee1d)
I20240501 20:11:32.456339 10735 raft_consensus.cc:504] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "596d3e24651b4787916397f573c09f1d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38865 } } peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:32.457371 10735 leader_election.cc:288] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865), 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397)
W20240501 20:11:32.458748 6481 leader_election.cc:334] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 596d3e24651b4787916397f573c09f1d (127.28.136.67:38865): Network error: Client connection negotiation failed: client connection to 127.28.136.67:38865: connect: Connection refused (error 111)
W20240501 20:11:32.458953 6483 leader_election.cc:334] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4acbc8659ce24119bd77571d1171ee1d (127.28.136.66:38397): Network error: Client connection negotiation failed: client connection to 127.28.136.66:38397: connect: Connection refused (error 111)
I20240501 20:11:32.459056 6483 leader_election.cc:302] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55fe2a4464a9402f927b2b60609b34dd; no voters: 4acbc8659ce24119bd77571d1171ee1d, 596d3e24651b4787916397f573c09f1d
I20240501 20:11:32.459308 10735 raft_consensus.cc:2726] T 28b9b57213a64172afdc2ca890569fc5 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:32.536602 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 6332
I20240501 20:11:32.560745 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 6465
I20240501 20:11:32.566304 6585 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:11:32.590703 7335 consensus_peers.cc:485] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 -> Peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Couldn't send request to peer 63c1e379865d42e78ea6b6b1d908ece7. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.633378 10772 raft_consensus.cc:482] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 63c1e379865d42e78ea6b6b1d908ece7)
I20240501 20:11:32.633630 10772 raft_consensus.cc:504] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "63c1e379865d42e78ea6b6b1d908ece7" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 43231 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.634433 10772 leader_election.cc:288] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231), 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971)
W20240501 20:11:32.635985 7335 leader_election.cc:334] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111)
W20240501 20:11:32.636188 7335 leader_election.cc:334] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9592068f16ca406eb8a347a65d365782 (127.28.136.71:40971): Network error: Client connection negotiation failed: client connection to 127.28.136.71:40971: connect: Connection refused (error 111)
I20240501 20:11:32.636296 7335 leader_election.cc:302] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b662c3da2b1b4f3693920ccb680b8d12; no voters: 63c1e379865d42e78ea6b6b1d908ece7, 9592068f16ca406eb8a347a65d365782
I20240501 20:11:32.636533 10772 raft_consensus.cc:2726] T 60871b6059c249bfaac713e2f7325811 P b662c3da2b1b4f3693920ccb680b8d12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:11:32.710191 10735 raft_consensus.cc:482] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4acbc8659ce24119bd77571d1171ee1d)
I20240501 20:11:32.710536 6546 raft_consensus.cc:1261] T eeb1eec290ce4dcbb2afedd57e3b81da P 55fe2a4464a9402f927b2b60609b34dd [term 3 FOLLOWER]: Refusing update from remote peer b662c3da2b1b4f3693920ccb680b8d12: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240501 20:11:32.710456 10735 raft_consensus.cc:504] T 79cb3f8b3fe94497bb54d3d8b7f18754 P 55fe2a4464a9402f927b2b60609b34dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "4acbc8659ce24119bd77571d1171ee1d" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 38397 } } peers { permanent_uuid: "9592068f16ca406eb8a347a65d365782" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 40971 } attrs { promote: false } } peers { permanent_uuid: "55fe2a4464a9402f927b2b60609b34dd" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 46003 } attrs { promote: false } }
I20240501 20:11:32.710698 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 6465
I20240501 20:11:32.733978 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 7234
W20240501 20:11:32.734405 7336 consensus_peers.cc:485] T eeb1eec290ce4dcbb2afedd57e3b81da P b662c3da2b1b4f3693920ccb680b8d12 -> Peer 55fe2a4464a9402f927b2b60609b34dd (127.28.136.72:46003): Couldn't send request to peer 55fe2a4464a9402f927b2b60609b34dd. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.72:46003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:11:32.735697 7335 consensus_peers.cc:485] T 3d7b1a47b1fe4e4b9a60b18b0b99056b P b662c3da2b1b4f3693920ccb680b8d12 -> Peer 63c1e379865d42e78ea6b6b1d908ece7 (127.28.136.65:43231): Couldn't send request to peer 63c1e379865d42e78ea6b6b1d908ece7. Status: Network error: Client connection negotiation failed: client connection to 127.28.136.65:43231: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:32.739979 7540 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:32.870683 10753 raft_consensus.cc:482] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 75b5acf03b9f4f6e98a67ae046587e79)
I20240501 20:11:32.870894 10753 raft_consensus.cc:504] T 5cdac8c0b92d442ebd84fb43af7e32fb P b662c3da2b1b4f3693920ccb680b8d12 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "75b5acf03b9f4f6e98a67ae046587e79" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36125 } } peers { permanent_uuid: "ee4daecd449e4fc981fd1fe2b0e492c2" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 45155 } attrs { promote: false } } peers { permanent_uuid: "b662c3da2b1b4f3693920ccb680b8d12" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 46209 } attrs { promote: false } }
I20240501 20:11:32.871253 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 7234
I20240501 20:11:32.891623 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 4001
I20240501 20:11:32.892719 4067 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:33.098497 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 4001
2024-05-01T20:11:33Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (51913 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (51913 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240501 20:11:33.251909 29217 test_util.cc:268] Using random seed: 399015702
I20240501 20:11:33.253960 29217 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:11:33.254065 29217 ts_itest-base.cc:113] --------------
I20240501 20:11:33.254113 29217 ts_itest-base.cc:114] 4 tablet servers
I20240501 20:11:33.254182 29217 ts_itest-base.cc:115] 1 replicas per TS
I20240501 20:11:33.254236 29217 ts_itest-base.cc:116] --------------
2024-05-01T20:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:11:33Z Disabled control of system clock
I20240501 20:11:33.316586 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:40137
--webserver_interface=127.28.136.126
--webserver_port=0
--builtin_ntp_servers=127.28.136.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.136.126:40137
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240501 20:11:33.669215 10784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:33.669562 10784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:33.669673 10784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:33.677512 10784 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:33.677655 10784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:11:33.677738 10784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:33.677810 10784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:11:33.677870 10784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:11:33.687515 10784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.28.136.126:40137
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:40137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.136.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:33.688951 10784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:33.690742 10784 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:33.701473 10789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:33.701687 10792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:33.702008 10790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:33.702726 10784 server_base.cc:1034] running on GCE node
I20240501 20:11:33.703545 10784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:33.705085 10784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:33.706298 10784 hybrid_clock.cc:648] HybridClock initialized: now 1714594293706247 us; error 42 us; skew 500 ppm
I20240501 20:11:33.706665 10784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:33.713503 10784 webserver.cc:460] Webserver started at http://127.28.136.126:38413/ using document root <none> and password file <none>
I20240501 20:11:33.714345 10784 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:33.714460 10784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:33.714782 10784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:33.717624 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7fbcc621d8f34f9cbeaa2c20678149f3"
format_stamp: "Formatted at 2024-05-01 20:11:33 on dist-test-slave-sx0n"
I20240501 20:11:33.718427 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7fbcc621d8f34f9cbeaa2c20678149f3"
format_stamp: "Formatted at 2024-05-01 20:11:33 on dist-test-slave-sx0n"
I20240501 20:11:33.723794 10784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:11:33.727317 10798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:33.728708 10784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:33.728921 10784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7fbcc621d8f34f9cbeaa2c20678149f3"
format_stamp: "Formatted at 2024-05-01 20:11:33 on dist-test-slave-sx0n"
I20240501 20:11:33.729141 10784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:33.743050 10784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:33.743958 10784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:33.744252 10784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:33.762576 10784 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.126:40137
I20240501 20:11:33.762595 10849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.126:40137 every 8 connection(s)
I20240501 20:11:33.764024 10784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:11:33.764794 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 10784
I20240501 20:11:33.765064 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:11:33.768497 10850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:33.780285 10850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3: Bootstrap starting.
I20240501 20:11:33.783398 10850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:33.784387 10850 log.cc:824] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:33.787091 10850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3: No bootstrap required, opened a new log
I20240501 20:11:33.793524 10850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbcc621d8f34f9cbeaa2c20678149f3" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 40137 } }
I20240501 20:11:33.793896 10850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:33.794016 10850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbcc621d8f34f9cbeaa2c20678149f3, State: Initialized, Role: FOLLOWER
I20240501 20:11:33.794620 10850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbcc621d8f34f9cbeaa2c20678149f3" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 40137 } }
I20240501 20:11:33.794818 10850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:33.794950 10850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:33.795102 10850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:33.797351 10850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbcc621d8f34f9cbeaa2c20678149f3" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 40137 } }
I20240501 20:11:33.797901 10850 leader_election.cc:302] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fbcc621d8f34f9cbeaa2c20678149f3; no voters:
I20240501 20:11:33.798480 10850 leader_election.cc:288] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:33.798607 10855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:33.799216 10855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [term 1 LEADER]: Becoming Leader. State: Replica: 7fbcc621d8f34f9cbeaa2c20678149f3, State: Running, Role: LEADER
I20240501 20:11:33.799750 10855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbcc621d8f34f9cbeaa2c20678149f3" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 40137 } }
I20240501 20:11:33.800472 10850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:11:33.802651 10857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fbcc621d8f34f9cbeaa2c20678149f3. Latest consensus state: current_term: 1 leader_uuid: "7fbcc621d8f34f9cbeaa2c20678149f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbcc621d8f34f9cbeaa2c20678149f3" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 40137 } } }
I20240501 20:11:33.802721 10856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fbcc621d8f34f9cbeaa2c20678149f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbcc621d8f34f9cbeaa2c20678149f3" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 40137 } } }
I20240501 20:11:33.803047 10857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:33.803054 10856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3 [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:33.806195 10862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:11:33.812397 10862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:11:33.821692 10862 catalog_manager.cc:1347] Generated new cluster ID: 83c55b5c636940ee88be4a85fea13680
I20240501 20:11:33.821851 10862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:11:33.868134 10862 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:11:33.869498 10862 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:11:33.887539 10862 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7fbcc621d8f34f9cbeaa2c20678149f3: Generated new TSK 0
I20240501 20:11:33.888474 10862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:11:33.909096 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.65:0
--local_ip_for_outbound_sockets=127.28.136.65
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:40137
--builtin_ntp_servers=127.28.136.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:34.270089 10874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:34.270413 10874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:34.270583 10874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:34.278451 10874 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:34.278585 10874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:34.278735 10874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.65
I20240501 20:11:34.288264 10874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:40137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.65
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:34.289742 10874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:34.291543 10874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:34.304033 10880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:34.304486 10879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:34.304950 10882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:34.305310 10874 server_base.cc:1034] running on GCE node
I20240501 20:11:34.305910 10874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:34.306849 10874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:34.308108 10874 hybrid_clock.cc:648] HybridClock initialized: now 1714594294308061 us; error 58 us; skew 500 ppm
I20240501 20:11:34.308400 10874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:34.310616 10874 webserver.cc:460] Webserver started at http://127.28.136.65:37643/ using document root <none> and password file <none>
I20240501 20:11:34.311300 10874 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:34.311396 10874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:34.311664 10874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:34.314787 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751"
format_stamp: "Formatted at 2024-05-01 20:11:34 on dist-test-slave-sx0n"
I20240501 20:11:34.315810 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751"
format_stamp: "Formatted at 2024-05-01 20:11:34 on dist-test-slave-sx0n"
I20240501 20:11:34.322834 10874 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20240501 20:11:34.327929 10888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:34.329842 10874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:34.330044 10874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751"
format_stamp: "Formatted at 2024-05-01 20:11:34 on dist-test-slave-sx0n"
I20240501 20:11:34.330332 10874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:34.352209 10874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:34.353241 10874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:34.353533 10874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:34.354506 10874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:34.356076 10874 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:34.356173 10874 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:34.356284 10874 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:34.356354 10874 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:34.386924 10874 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.65:39213
I20240501 20:11:34.386934 11000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.65:39213 every 8 connection(s)
I20240501 20:11:34.388491 10874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:11:34.391788 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 10874
I20240501 20:11:34.392143 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:11:34.396030 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.66:0
--local_ip_for_outbound_sockets=127.28.136.66
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:40137
--builtin_ntp_servers=127.28.136.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:34.402875 11001 heartbeater.cc:347] Connected to a master server at 127.28.136.126:40137
I20240501 20:11:34.403205 11001 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:34.404091 11001 heartbeater.cc:511] Master 127.28.136.126:40137 requested a full tablet report, sending...
I20240501 20:11:34.406277 10815 ts_manager.cc:194] Registered new tserver with Master: ae8ee11327ae4e0b90c1b8c5e1bcb751 (127.28.136.65:39213)
I20240501 20:11:34.407912 10815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.65:56309
W20240501 20:11:34.753650 11005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:34.753968 11005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:34.754084 11005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:34.761607 11005 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:34.761729 11005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:34.761884 11005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.66
I20240501 20:11:34.771078 11005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:40137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.66
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:34.772398 11005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:34.774121 11005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:34.786296 11013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:34.786648 11010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:34.787110 11011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:34.787770 11005 server_base.cc:1034] running on GCE node
I20240501 20:11:34.788408 11005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:34.789467 11005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:34.790664 11005 hybrid_clock.cc:648] HybridClock initialized: now 1714594294790614 us; error 40 us; skew 500 ppm
I20240501 20:11:34.790985 11005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:34.793340 11005 webserver.cc:460] Webserver started at http://127.28.136.66:43461/ using document root <none> and password file <none>
I20240501 20:11:34.794087 11005 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:34.794190 11005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:34.794503 11005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:34.797176 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "220e903189f440beb7ac7a751dc00f26"
format_stamp: "Formatted at 2024-05-01 20:11:34 on dist-test-slave-sx0n"
I20240501 20:11:34.797960 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "220e903189f440beb7ac7a751dc00f26"
format_stamp: "Formatted at 2024-05-01 20:11:34 on dist-test-slave-sx0n"
I20240501 20:11:34.803211 11005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:11:34.806597 11019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:34.807897 11005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:34.808056 11005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "220e903189f440beb7ac7a751dc00f26"
format_stamp: "Formatted at 2024-05-01 20:11:34 on dist-test-slave-sx0n"
I20240501 20:11:34.808247 11005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:34.822782 11005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:34.823575 11005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:34.823839 11005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:34.824786 11005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:34.826238 11005 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:34.826332 11005 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:34.826444 11005 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:34.826520 11005 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:34.857551 11005 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.66:46771
I20240501 20:11:34.857597 11131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.66:46771 every 8 connection(s)
I20240501 20:11:34.858870 11005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:11:34.862831 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 11005
I20240501 20:11:34.863034 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:11:34.866845 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:0
--local_ip_for_outbound_sockets=127.28.136.67
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:40137
--builtin_ntp_servers=127.28.136.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:34.872856 11132 heartbeater.cc:347] Connected to a master server at 127.28.136.126:40137
I20240501 20:11:34.873196 11132 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:34.874217 11132 heartbeater.cc:511] Master 127.28.136.126:40137 requested a full tablet report, sending...
I20240501 20:11:34.875880 10815 ts_manager.cc:194] Registered new tserver with Master: 220e903189f440beb7ac7a751dc00f26 (127.28.136.66:46771)
I20240501 20:11:34.876922 10815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.66:57877
W20240501 20:11:35.212283 11135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:35.212633 11135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:35.212791 11135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:35.220315 11135 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:35.220441 11135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:35.220611 11135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:11:35.229840 11135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:40137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:35.231238 11135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:35.233070 11135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:35.245546 11140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:35.245945 11141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:35.246531 11135 server_base.cc:1034] running on GCE node
W20240501 20:11:35.246893 11143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:35.247608 11135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:35.248600 11135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:35.249809 11135 hybrid_clock.cc:648] HybridClock initialized: now 1714594295249739 us; error 80 us; skew 500 ppm
I20240501 20:11:35.250135 11135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:35.252388 11135 webserver.cc:460] Webserver started at http://127.28.136.67:40975/ using document root <none> and password file <none>
I20240501 20:11:35.253212 11135 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:35.253334 11135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:35.253639 11135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:35.256237 11135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "deb94eef62fc494fa1286ef07edb0f55"
format_stamp: "Formatted at 2024-05-01 20:11:35 on dist-test-slave-sx0n"
I20240501 20:11:35.257067 11135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "deb94eef62fc494fa1286ef07edb0f55"
format_stamp: "Formatted at 2024-05-01 20:11:35 on dist-test-slave-sx0n"
I20240501 20:11:35.262225 11135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:11:35.265663 11149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:35.267060 11135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:11:35.267215 11135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "deb94eef62fc494fa1286ef07edb0f55"
format_stamp: "Formatted at 2024-05-01 20:11:35 on dist-test-slave-sx0n"
I20240501 20:11:35.267412 11135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:35.293109 11135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:35.293990 11135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:35.294261 11135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:35.295254 11135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:35.296751 11135 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:35.296847 11135 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:35.296963 11135 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:35.297032 11135 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:35.331797 11135 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:34987
I20240501 20:11:35.331844 11261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:34987 every 8 connection(s)
I20240501 20:11:35.333212 11135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:35.335330 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 11135
I20240501 20:11:35.335517 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:11:35.339176 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.68:0
--local_ip_for_outbound_sockets=127.28.136.68
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:40137
--builtin_ntp_servers=127.28.136.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:35.347961 11262 heartbeater.cc:347] Connected to a master server at 127.28.136.126:40137
I20240501 20:11:35.348315 11262 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:35.349356 11262 heartbeater.cc:511] Master 127.28.136.126:40137 requested a full tablet report, sending...
I20240501 20:11:35.351023 10815 ts_manager.cc:194] Registered new tserver with Master: deb94eef62fc494fa1286ef07edb0f55 (127.28.136.67:34987)
I20240501 20:11:35.352092 10815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:52255
I20240501 20:11:35.410990 11001 heartbeater.cc:503] Master 127.28.136.126:40137 was elected leader, sending a full tablet report...
W20240501 20:11:35.682720 11265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:35.683049 11265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:35.683187 11265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:35.690719 11265 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:35.690868 11265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:35.691047 11265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.68
I20240501 20:11:35.700189 11265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:40137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.68
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:35.701633 11265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:35.703343 11265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:35.715065 11270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:35.715286 11273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:35.715811 11271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:35.716393 11265 server_base.cc:1034] running on GCE node
I20240501 20:11:35.717000 11265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:35.717883 11265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:35.719077 11265 hybrid_clock.cc:648] HybridClock initialized: now 1714594295719028 us; error 39 us; skew 500 ppm
I20240501 20:11:35.719377 11265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:35.721539 11265 webserver.cc:460] Webserver started at http://127.28.136.68:38739/ using document root <none> and password file <none>
I20240501 20:11:35.722261 11265 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:35.722373 11265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:35.722681 11265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:35.725258 11265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "add66495a78243b4af7fd39722c62782"
format_stamp: "Formatted at 2024-05-01 20:11:35 on dist-test-slave-sx0n"
I20240501 20:11:35.726011 11265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "add66495a78243b4af7fd39722c62782"
format_stamp: "Formatted at 2024-05-01 20:11:35 on dist-test-slave-sx0n"
I20240501 20:11:35.730957 11265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:11:35.734174 11279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:35.735571 11265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:11:35.735726 11265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "add66495a78243b4af7fd39722c62782"
format_stamp: "Formatted at 2024-05-01 20:11:35 on dist-test-slave-sx0n"
I20240501 20:11:35.735919 11265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:35.773150 11265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:35.774050 11265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:35.774327 11265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:35.775312 11265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:35.776878 11265 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:35.776978 11265 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:35.777091 11265 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:35.777163 11265 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:35.806519 11265 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.68:33523
I20240501 20:11:35.806586 11391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.68:33523 every 8 connection(s)
I20240501 20:11:35.807917 11265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:35.809125 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 11265
I20240501 20:11:35.809355 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:11:35.820040 11392 heartbeater.cc:347] Connected to a master server at 127.28.136.126:40137
I20240501 20:11:35.820322 11392 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:35.821135 11392 heartbeater.cc:511] Master 127.28.136.126:40137 requested a full tablet report, sending...
I20240501 20:11:35.822412 10815 ts_manager.cc:194] Registered new tserver with Master: add66495a78243b4af7fd39722c62782 (127.28.136.68:33523)
I20240501 20:11:35.823180 10815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.68:47427
I20240501 20:11:35.825484 29217 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:11:35.840664 10815 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46472:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:11:35.868546 10936 tablet_service.cc:1467] Processing CreateTablet for tablet 80caff75bad24692882671275b46c6ff (DEFAULT_TABLE table=TestTable [id=385c385f052d4143b9f6f0d90418655f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:35.869890 10936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80caff75bad24692882671275b46c6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:35.879850 11406 tablet_bootstrap.cc:492] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751: Bootstrap starting.
I20240501 20:11:35.879931 11132 heartbeater.cc:503] Master 127.28.136.126:40137 was elected leader, sending a full tablet report...
I20240501 20:11:35.883129 11406 tablet_bootstrap.cc:654] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:35.884133 11406 log.cc:824] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:35.886492 11406 tablet_bootstrap.cc:492] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751: No bootstrap required, opened a new log
I20240501 20:11:35.886736 11406 ts_tablet_manager.cc:1367] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:11:35.892860 11406 raft_consensus.cc:348] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:35.893177 11406 raft_consensus.cc:374] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:35.893277 11406 raft_consensus.cc:729] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Initialized, Role: FOLLOWER
I20240501 20:11:35.893893 11406 consensus_queue.cc:260] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:35.894093 11406 raft_consensus.cc:388] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:35.894215 11406 raft_consensus.cc:482] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:35.894359 11406 raft_consensus.cc:3037] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:35.896257 11406 raft_consensus.cc:504] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:35.896821 11406 leader_election.cc:302] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8ee11327ae4e0b90c1b8c5e1bcb751; no voters:
I20240501 20:11:35.897531 11406 leader_election.cc:288] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:35.897679 11408 raft_consensus.cc:2781] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:35.898239 11408 raft_consensus.cc:686] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 LEADER]: Becoming Leader. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Running, Role: LEADER
I20240501 20:11:35.899068 11408 consensus_queue.cc:237] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:35.899390 11406 ts_tablet_manager.cc:1398] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20240501 20:11:35.904418 10814 catalog_manager.cc:5484] T 80caff75bad24692882671275b46c6ff P ae8ee11327ae4e0b90c1b8c5e1bcb751 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8ee11327ae4e0b90c1b8c5e1bcb751 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:35.913537 29217 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:11:35.915313 29217 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ae8ee11327ae4e0b90c1b8c5e1bcb751 to finish bootstrapping
I20240501 20:11:35.923548 29217 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 220e903189f440beb7ac7a751dc00f26 to finish bootstrapping
I20240501 20:11:35.933403 29217 ts_itest-base.cc:233] Waiting for 0 tablets on tserver deb94eef62fc494fa1286ef07edb0f55 to finish bootstrapping
I20240501 20:11:35.941387 29217 ts_itest-base.cc:233] Waiting for 0 tablets on tserver add66495a78243b4af7fd39722c62782 to finish bootstrapping
I20240501 20:11:35.949491 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 11135
I20240501 20:11:35.950353 11257 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:36.028861 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 11135
I20240501 20:11:36.043502 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 11265
I20240501 20:11:36.044633 11387 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:36.124361 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 11265
I20240501 20:11:39.890044 10814 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46472:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:39.900486 10936 tablet_service.cc:1467] Processing CreateTablet for tablet dcc24131a5e047519a6cab0a96f3b914 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4980e575c90742cba3b8153cf2210552]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.901011 10936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc24131a5e047519a6cab0a96f3b914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.906952 11425 tablet_bootstrap.cc:492] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Bootstrap starting.
I20240501 20:11:39.908799 11425 tablet_bootstrap.cc:654] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.909399 11066 tablet_service.cc:1467] Processing CreateTablet for tablet 60930f851c3146c980f69309684dd9c9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4980e575c90742cba3b8153cf2210552]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.909350 11067 tablet_service.cc:1467] Processing CreateTablet for tablet 70a872401e3541ed81dfbb47b52877cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=4980e575c90742cba3b8153cf2210552]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:39.910763 11067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70a872401e3541ed81dfbb47b52877cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.910868 11425 tablet_bootstrap.cc:492] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751: No bootstrap required, opened a new log
I20240501 20:11:39.911063 11425 ts_tablet_manager.cc:1367] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:11:39.911463 11066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60930f851c3146c980f69309684dd9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:39.911911 11425 raft_consensus.cc:348] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:39.912056 11425 raft_consensus.cc:374] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.912175 11425 raft_consensus.cc:729] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.912503 11425 consensus_queue.cc:260] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:39.912911 11425 raft_consensus.cc:388] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:39.913028 11425 raft_consensus.cc:482] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:39.913154 11425 raft_consensus.cc:3037] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.914961 11425 raft_consensus.cc:504] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:39.915177 11425 leader_election.cc:302] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8ee11327ae4e0b90c1b8c5e1bcb751; no voters:
I20240501 20:11:39.915632 11425 leader_election.cc:288] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:39.915774 11427 raft_consensus.cc:2781] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:39.916807 11425 ts_tablet_manager.cc:1398] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240501 20:11:39.922472 11430 tablet_bootstrap.cc:492] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26: Bootstrap starting.
I20240501 20:11:39.924974 11427 raft_consensus.cc:686] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 LEADER]: Becoming Leader. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Running, Role: LEADER
I20240501 20:11:39.925323 11427 consensus_queue.cc:237] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:39.926050 11430 tablet_bootstrap.cc:654] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.927383 11430 log.cc:824] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:39.928860 10814 catalog_manager.cc:5484] T dcc24131a5e047519a6cab0a96f3b914 P ae8ee11327ae4e0b90c1b8c5e1bcb751 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8ee11327ae4e0b90c1b8c5e1bcb751 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:39.930262 11430 tablet_bootstrap.cc:492] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26: No bootstrap required, opened a new log
I20240501 20:11:39.930519 11430 ts_tablet_manager.cc:1367] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240501 20:11:39.937494 11430 raft_consensus.cc:348] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:39.937888 11430 raft_consensus.cc:374] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.937994 11430 raft_consensus.cc:729] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.938648 11430 consensus_queue.cc:260] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:39.938863 11430 raft_consensus.cc:388] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:39.938992 11430 raft_consensus.cc:482] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:39.939143 11430 raft_consensus.cc:3037] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.941345 11430 raft_consensus.cc:504] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:39.941823 11430 leader_election.cc:302] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220e903189f440beb7ac7a751dc00f26; no voters:
I20240501 20:11:39.942430 11430 leader_election.cc:288] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:39.942589 11434 raft_consensus.cc:2781] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:39.943153 11434 raft_consensus.cc:686] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [term 1 LEADER]: Becoming Leader. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Running, Role: LEADER
I20240501 20:11:39.943817 11434 consensus_queue.cc:237] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:39.944093 11430 ts_tablet_manager.cc:1398] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20240501 20:11:39.944439 11430 tablet_bootstrap.cc:492] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26: Bootstrap starting.
I20240501 20:11:39.946069 11430 tablet_bootstrap.cc:654] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:39.948294 11430 tablet_bootstrap.cc:492] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26: No bootstrap required, opened a new log
I20240501 20:11:39.948450 11430 ts_tablet_manager.cc:1367] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:39.949111 10814 catalog_manager.cc:5484] T 70a872401e3541ed81dfbb47b52877cd P 220e903189f440beb7ac7a751dc00f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220e903189f440beb7ac7a751dc00f26 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "220e903189f440beb7ac7a751dc00f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:39.949460 11430 raft_consensus.cc:348] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:39.949638 11430 raft_consensus.cc:374] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:39.949739 11430 raft_consensus.cc:729] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Initialized, Role: FOLLOWER
I20240501 20:11:39.949995 11430 consensus_queue.cc:260] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:39.950184 11430 raft_consensus.cc:388] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:39.950264 11430 raft_consensus.cc:482] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:39.950361 11430 raft_consensus.cc:3037] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:39.952652 11430 raft_consensus.cc:504] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:39.952920 11430 leader_election.cc:302] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220e903189f440beb7ac7a751dc00f26; no voters:
I20240501 20:11:39.953119 11430 leader_election.cc:288] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:39.953182 11434 raft_consensus.cc:2781] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:39.953375 11434 raft_consensus.cc:686] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [term 1 LEADER]: Becoming Leader. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Running, Role: LEADER
I20240501 20:11:39.953576 11430 ts_tablet_manager.cc:1398] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:11:39.953581 11434 consensus_queue.cc:237] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:39.956301 10814 catalog_manager.cc:5484] T 60930f851c3146c980f69309684dd9c9 P 220e903189f440beb7ac7a751dc00f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220e903189f440beb7ac7a751dc00f26 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "220e903189f440beb7ac7a751dc00f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:40.474234 10814 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46472:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:40.482697 11066 tablet_service.cc:1467] Processing CreateTablet for tablet c4a6bd894ba64689bb7c0fa5c5a48d9b (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2dd16b92a2941278aa73dca195189bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.482668 10936 tablet_service.cc:1467] Processing CreateTablet for tablet 5260ffd7948f41099f4e9017dd7a5fd0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2dd16b92a2941278aa73dca195189bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.483006 10935 tablet_service.cc:1467] Processing CreateTablet for tablet fcd6b4412cad4573b5915989019ee0cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=b2dd16b92a2941278aa73dca195189bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:40.483141 11066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a6bd894ba64689bb7c0fa5c5a48d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.483152 10936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5260ffd7948f41099f4e9017dd7a5fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.483345 10935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcd6b4412cad4573b5915989019ee0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:40.489259 11464 tablet_bootstrap.cc:492] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751: Bootstrap starting.
I20240501 20:11:40.490760 11464 tablet_bootstrap.cc:654] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.491165 11465 tablet_bootstrap.cc:492] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26: Bootstrap starting.
I20240501 20:11:40.492759 11464 tablet_bootstrap.cc:492] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751: No bootstrap required, opened a new log
I20240501 20:11:40.492789 11465 tablet_bootstrap.cc:654] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.492911 11464 ts_tablet_manager.cc:1367] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 20:11:40.493706 11464 raft_consensus.cc:348] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:40.493880 11464 raft_consensus.cc:374] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.493973 11464 raft_consensus.cc:729] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.494222 11464 consensus_queue.cc:260] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:40.494418 11464 raft_consensus.cc:388] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:40.494493 11464 raft_consensus.cc:482] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:40.494593 11464 raft_consensus.cc:3037] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.494745 11465 tablet_bootstrap.cc:492] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26: No bootstrap required, opened a new log
I20240501 20:11:40.494889 11465 ts_tablet_manager.cc:1367] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:40.495609 11465 raft_consensus.cc:348] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:40.495784 11465 raft_consensus.cc:374] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.495872 11465 raft_consensus.cc:729] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.496079 11465 consensus_queue.cc:260] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:40.496279 11465 raft_consensus.cc:388] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:40.496361 11465 raft_consensus.cc:482] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:40.496460 11465 raft_consensus.cc:3037] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.497215 11464 raft_consensus.cc:504] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:40.497475 11464 leader_election.cc:302] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8ee11327ae4e0b90c1b8c5e1bcb751; no voters:
I20240501 20:11:40.497892 11464 leader_election.cc:288] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:40.498064 11468 raft_consensus.cc:2781] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:40.498575 11464 ts_tablet_manager.cc:1398] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240501 20:11:40.498935 11464 tablet_bootstrap.cc:492] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Bootstrap starting.
I20240501 20:11:40.499080 11465 raft_consensus.cc:504] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:40.499382 11465 leader_election.cc:302] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220e903189f440beb7ac7a751dc00f26; no voters:
I20240501 20:11:40.499745 11434 raft_consensus.cc:2781] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:40.500177 11434 raft_consensus.cc:686] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [term 1 LEADER]: Becoming Leader. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Running, Role: LEADER
I20240501 20:11:40.500444 11434 consensus_queue.cc:237] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:40.500715 11464 tablet_bootstrap.cc:654] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:40.500903 11465 leader_election.cc:288] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:40.501535 11465 ts_tablet_manager.cc:1398] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240501 20:11:40.502786 11468 raft_consensus.cc:686] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 LEADER]: Becoming Leader. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Running, Role: LEADER
I20240501 20:11:40.502966 11464 tablet_bootstrap.cc:492] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751: No bootstrap required, opened a new log
I20240501 20:11:40.503154 11464 ts_tablet_manager.cc:1367] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:11:40.503178 11468 consensus_queue.cc:237] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:40.504006 11464 raft_consensus.cc:348] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:40.503993 10814 catalog_manager.cc:5484] T c4a6bd894ba64689bb7c0fa5c5a48d9b P 220e903189f440beb7ac7a751dc00f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220e903189f440beb7ac7a751dc00f26 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "220e903189f440beb7ac7a751dc00f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:40.504235 11464 raft_consensus.cc:374] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:40.504438 11464 raft_consensus.cc:729] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Initialized, Role: FOLLOWER
I20240501 20:11:40.504807 11464 consensus_queue.cc:260] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:40.505059 11464 raft_consensus.cc:388] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:40.505205 11464 raft_consensus.cc:482] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:40.505347 11464 raft_consensus.cc:3037] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:40.506397 10815 catalog_manager.cc:5484] T fcd6b4412cad4573b5915989019ee0cc P ae8ee11327ae4e0b90c1b8c5e1bcb751 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8ee11327ae4e0b90c1b8c5e1bcb751 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:40.508817 11464 raft_consensus.cc:504] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:40.509148 11464 leader_election.cc:302] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8ee11327ae4e0b90c1b8c5e1bcb751; no voters:
I20240501 20:11:40.509428 11464 leader_election.cc:288] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:40.509516 11470 raft_consensus.cc:2781] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:40.510172 11464 ts_tablet_manager.cc:1398] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:11:40.510129 11470 raft_consensus.cc:686] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 LEADER]: Becoming Leader. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Running, Role: LEADER
I20240501 20:11:40.510483 11470 consensus_queue.cc:237] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:40.513528 10815 catalog_manager.cc:5484] T 5260ffd7948f41099f4e9017dd7a5fd0 P ae8ee11327ae4e0b90c1b8c5e1bcb751 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8ee11327ae4e0b90c1b8c5e1bcb751 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:41.041962 10815 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46472:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:41.050877 11066 tablet_service.cc:1467] Processing CreateTablet for tablet 5fce0bc3a7b2433a9fc82cbdf389a2b8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f66256eb0274342a8cae5c042cb9ad6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.051209 11067 tablet_service.cc:1467] Processing CreateTablet for tablet 40d8a175340b49959801789ad7049ef2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f66256eb0274342a8cae5c042cb9ad6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.051178 10935 tablet_service.cc:1467] Processing CreateTablet for tablet 76c984ada15047fe9110dc8ef4bedae6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9f66256eb0274342a8cae5c042cb9ad6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:41.051378 11066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fce0bc3a7b2433a9fc82cbdf389a2b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.051616 11067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d8a175340b49959801789ad7049ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.051640 10935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c984ada15047fe9110dc8ef4bedae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:41.056677 11502 tablet_bootstrap.cc:492] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26: Bootstrap starting.
I20240501 20:11:41.058252 11502 tablet_bootstrap.cc:654] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.058679 11503 tablet_bootstrap.cc:492] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Bootstrap starting.
I20240501 20:11:41.060595 11502 tablet_bootstrap.cc:492] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26: No bootstrap required, opened a new log
I20240501 20:11:41.060627 11503 tablet_bootstrap.cc:654] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.060766 11502 ts_tablet_manager.cc:1367] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:41.061398 11502 raft_consensus.cc:348] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:41.061565 11502 raft_consensus.cc:374] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.061664 11502 raft_consensus.cc:729] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.061904 11502 consensus_queue.cc:260] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:41.062080 11502 raft_consensus.cc:388] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:41.062196 11502 raft_consensus.cc:482] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:41.062314 11502 raft_consensus.cc:3037] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.062397 11503 tablet_bootstrap.cc:492] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751: No bootstrap required, opened a new log
I20240501 20:11:41.062547 11503 ts_tablet_manager.cc:1367] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 20:11:41.063261 11503 raft_consensus.cc:348] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:41.063442 11503 raft_consensus.cc:374] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.063585 11503 raft_consensus.cc:729] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.063843 11503 consensus_queue.cc:260] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:41.064062 11503 raft_consensus.cc:388] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:41.064143 11503 raft_consensus.cc:482] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:41.064250 11503 raft_consensus.cc:3037] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.064653 11502 raft_consensus.cc:504] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:41.064929 11502 leader_election.cc:302] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220e903189f440beb7ac7a751dc00f26; no voters:
I20240501 20:11:41.065318 11502 leader_election.cc:288] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:41.065513 11506 raft_consensus.cc:2781] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:41.066040 11502 ts_tablet_manager.cc:1398] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240501 20:11:41.066293 11502 tablet_bootstrap.cc:492] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26: Bootstrap starting.
I20240501 20:11:41.067152 11503 raft_consensus.cc:504] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:41.067530 11503 leader_election.cc:302] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8ee11327ae4e0b90c1b8c5e1bcb751; no voters:
I20240501 20:11:41.067858 11470 raft_consensus.cc:2781] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:41.067950 11502 tablet_bootstrap.cc:654] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:41.072315 11506 raft_consensus.cc:686] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [term 1 LEADER]: Becoming Leader. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Running, Role: LEADER
I20240501 20:11:41.072762 11506 consensus_queue.cc:237] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:41.073073 11503 leader_election.cc:288] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:41.074715 11470 raft_consensus.cc:686] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 LEADER]: Becoming Leader. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Running, Role: LEADER
I20240501 20:11:41.075129 11470 consensus_queue.cc:237] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:41.076668 11502 tablet_bootstrap.cc:492] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26: No bootstrap required, opened a new log
I20240501 20:11:41.076903 11502 ts_tablet_manager.cc:1367] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 20:11:41.077767 11502 raft_consensus.cc:348] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:41.077991 11502 raft_consensus.cc:374] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:41.078088 11502 raft_consensus.cc:729] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Initialized, Role: FOLLOWER
I20240501 20:11:41.078352 11502 consensus_queue.cc:260] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:41.078578 11502 raft_consensus.cc:388] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:41.078687 11502 raft_consensus.cc:482] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:41.078792 11502 raft_consensus.cc:3037] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:41.081049 11503 ts_tablet_manager.cc:1398] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent starting tablet: real 0.018s user 0.004s sys 0.008s
I20240501 20:11:41.082489 11502 raft_consensus.cc:504] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:41.082828 11502 leader_election.cc:302] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220e903189f440beb7ac7a751dc00f26; no voters:
I20240501 20:11:41.083079 11502 leader_election.cc:288] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:41.083006 10815 catalog_manager.cc:5484] T 5fce0bc3a7b2433a9fc82cbdf389a2b8 P 220e903189f440beb7ac7a751dc00f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220e903189f440beb7ac7a751dc00f26 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "220e903189f440beb7ac7a751dc00f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:41.083472 10814 catalog_manager.cc:5484] T 76c984ada15047fe9110dc8ef4bedae6 P ae8ee11327ae4e0b90c1b8c5e1bcb751 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8ee11327ae4e0b90c1b8c5e1bcb751 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:41.084646 11508 raft_consensus.cc:2781] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:41.084844 11508 raft_consensus.cc:686] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [term 1 LEADER]: Becoming Leader. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Running, Role: LEADER
I20240501 20:11:41.084993 11502 ts_tablet_manager.cc:1398] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20240501 20:11:41.085188 11508 consensus_queue.cc:237] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:41.089274 10814 catalog_manager.cc:5484] T 40d8a175340b49959801789ad7049ef2 P 220e903189f440beb7ac7a751dc00f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220e903189f440beb7ac7a751dc00f26 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "220e903189f440beb7ac7a751dc00f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:42.450518 10813 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46472:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:42.459641 10935 tablet_service.cc:1467] Processing CreateTablet for tablet d44e00df30a04719bb7015cb54326eb5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=10d3830655894c57a58a887608a5478e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:42.460177 10935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44e00df30a04719bb7015cb54326eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:42.460183 10936 tablet_service.cc:1467] Processing CreateTablet for tablet 93e20193673049e3a1242470915fa790 (DEFAULT_TABLE table=rebalance_test_table_3 [id=10d3830655894c57a58a887608a5478e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:42.460661 11067 tablet_service.cc:1467] Processing CreateTablet for tablet b3cc1b4bbd874be090c8eb9e2f2a0f65 (DEFAULT_TABLE table=rebalance_test_table_3 [id=10d3830655894c57a58a887608a5478e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:42.460706 10936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e20193673049e3a1242470915fa790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:42.461107 11067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3cc1b4bbd874be090c8eb9e2f2a0f65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:42.466799 11543 tablet_bootstrap.cc:492] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Bootstrap starting.
I20240501 20:11:42.467329 11544 tablet_bootstrap.cc:492] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26: Bootstrap starting.
I20240501 20:11:42.468875 11544 tablet_bootstrap.cc:654] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:42.468878 11543 tablet_bootstrap.cc:654] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:42.470806 11544 tablet_bootstrap.cc:492] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26: No bootstrap required, opened a new log
I20240501 20:11:42.470806 11543 tablet_bootstrap.cc:492] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751: No bootstrap required, opened a new log
I20240501 20:11:42.470948 11544 ts_tablet_manager.cc:1367] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:11:42.470948 11543 ts_tablet_manager.cc:1367] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:11:42.471694 11544 raft_consensus.cc:348] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:42.471694 11543 raft_consensus.cc:348] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:42.471875 11543 raft_consensus.cc:374] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:42.471875 11544 raft_consensus.cc:374] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:42.471969 11544 raft_consensus.cc:729] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Initialized, Role: FOLLOWER
I20240501 20:11:42.471978 11543 raft_consensus.cc:729] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Initialized, Role: FOLLOWER
I20240501 20:11:42.472188 11544 consensus_queue.cc:260] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:42.472203 11543 consensus_queue.cc:260] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:42.472375 11543 raft_consensus.cc:388] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:42.472404 11544 raft_consensus.cc:388] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:42.472468 11543 raft_consensus.cc:482] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:42.472497 11544 raft_consensus.cc:482] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:42.472580 11543 raft_consensus.cc:3037] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:42.472640 11544 raft_consensus.cc:3037] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:42.475320 11543 raft_consensus.cc:504] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:42.475299 11544 raft_consensus.cc:504] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:42.475574 11544 leader_election.cc:302] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 220e903189f440beb7ac7a751dc00f26; no voters:
I20240501 20:11:42.475574 11543 leader_election.cc:302] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8ee11327ae4e0b90c1b8c5e1bcb751; no voters:
I20240501 20:11:42.475997 11544 leader_election.cc:288] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:42.476137 11548 raft_consensus.cc:2781] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:42.476624 11543 leader_election.cc:288] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:42.476786 11547 raft_consensus.cc:2781] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:42.477699 11548 raft_consensus.cc:686] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [term 1 LEADER]: Becoming Leader. State: Replica: 220e903189f440beb7ac7a751dc00f26, State: Running, Role: LEADER
I20240501 20:11:42.477981 11548 consensus_queue.cc:237] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } }
I20240501 20:11:42.480428 10813 catalog_manager.cc:5484] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 220e903189f440beb7ac7a751dc00f26 (127.28.136.66). New cstate: current_term: 1 leader_uuid: "220e903189f440beb7ac7a751dc00f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "220e903189f440beb7ac7a751dc00f26" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 46771 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:42.480854 11543 ts_tablet_manager.cc:1398] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20240501 20:11:42.481156 11543 tablet_bootstrap.cc:492] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Bootstrap starting.
I20240501 20:11:42.481734 11547 raft_consensus.cc:686] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 LEADER]: Becoming Leader. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Running, Role: LEADER
I20240501 20:11:42.482019 11547 consensus_queue.cc:237] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:42.485564 11543 tablet_bootstrap.cc:654] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:42.487262 11544 ts_tablet_manager.cc:1398] T b3cc1b4bbd874be090c8eb9e2f2a0f65 P 220e903189f440beb7ac7a751dc00f26: Time spent starting tablet: real 0.016s user 0.000s sys 0.016s
I20240501 20:11:42.489964 11543 tablet_bootstrap.cc:492] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751: No bootstrap required, opened a new log
I20240501 20:11:42.490142 11543 ts_tablet_manager.cc:1367] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:11:42.490204 10814 catalog_manager.cc:5484] T d44e00df30a04719bb7015cb54326eb5 P ae8ee11327ae4e0b90c1b8c5e1bcb751 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8ee11327ae4e0b90c1b8c5e1bcb751 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:42.491075 11543 raft_consensus.cc:348] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:42.491248 11543 raft_consensus.cc:374] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:42.491318 11543 raft_consensus.cc:729] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Initialized, Role: FOLLOWER
I20240501 20:11:42.491587 11543 consensus_queue.cc:260] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:42.491771 11543 raft_consensus.cc:388] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:42.491881 11543 raft_consensus.cc:482] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:42.491988 11543 raft_consensus.cc:3037] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:42.494403 11543 raft_consensus.cc:504] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:42.494602 11543 leader_election.cc:302] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8ee11327ae4e0b90c1b8c5e1bcb751; no voters:
I20240501 20:11:42.494769 11543 leader_election.cc:288] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:42.494884 11552 raft_consensus.cc:2781] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:42.495296 11543 ts_tablet_manager.cc:1398] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:11:42.495517 11552 raft_consensus.cc:686] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [term 1 LEADER]: Becoming Leader. State: Replica: ae8ee11327ae4e0b90c1b8c5e1bcb751, State: Running, Role: LEADER
I20240501 20:11:42.495765 11552 consensus_queue.cc:237] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } }
I20240501 20:11:42.504520 10813 catalog_manager.cc:5484] T 93e20193673049e3a1242470915fa790 P ae8ee11327ae4e0b90c1b8c5e1bcb751 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8ee11327ae4e0b90c1b8c5e1bcb751 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ee11327ae4e0b90c1b8c5e1bcb751" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 39213 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:43.005097 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:34987
--local_ip_for_outbound_sockets=127.28.136.67
--tserver_master_addrs=127.28.136.126:40137
--webserver_port=40975
--webserver_interface=127.28.136.67
--builtin_ntp_servers=127.28.136.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:43.342693 11583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:43.343000 11583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:43.343134 11583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:43.350648 11583 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:43.350787 11583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:43.350951 11583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:11:43.360137 11583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:34987
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=40975
--tserver_master_addrs=127.28.136.126:40137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:43.361521 11583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:43.363309 11583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:43.375490 11588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:43.375937 11589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:43.376263 11591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:43.376721 11583 server_base.cc:1034] running on GCE node
I20240501 20:11:43.377406 11583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:43.378360 11583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:43.379565 11583 hybrid_clock.cc:648] HybridClock initialized: now 1714594303379514 us; error 38 us; skew 500 ppm
I20240501 20:11:43.379909 11583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:43.382120 11583 webserver.cc:460] Webserver started at http://127.28.136.67:40975/ using document root <none> and password file <none>
I20240501 20:11:43.382848 11583 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:43.382979 11583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:43.388366 11583 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.001s
I20240501 20:11:43.391292 11597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.392688 11583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:43.392864 11583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "deb94eef62fc494fa1286ef07edb0f55"
format_stamp: "Formatted at 2024-05-01 20:11:35 on dist-test-slave-sx0n"
I20240501 20:11:43.393674 11583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:43.422569 11583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:43.423568 11583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:43.423874 11583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:43.424870 11583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:43.426437 11583 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:43.426517 11583 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.426641 11583 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:43.426694 11583 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.456264 11583 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:34987
I20240501 20:11:43.456310 11709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:34987 every 8 connection(s)
I20240501 20:11:43.457749 11583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:43.459142 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 11583
I20240501 20:11:43.459941 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.68:33523
--local_ip_for_outbound_sockets=127.28.136.68
--tserver_master_addrs=127.28.136.126:40137
--webserver_port=38739
--webserver_interface=127.28.136.68
--builtin_ntp_servers=127.28.136.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:43.471943 11710 heartbeater.cc:347] Connected to a master server at 127.28.136.126:40137
I20240501 20:11:43.472295 11710 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:43.473313 11710 heartbeater.cc:511] Master 127.28.136.126:40137 requested a full tablet report, sending...
I20240501 20:11:43.474944 10813 ts_manager.cc:194] Re-registered known tserver with Master: deb94eef62fc494fa1286ef07edb0f55 (127.28.136.67:34987)
I20240501 20:11:43.476055 10813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:50753
W20240501 20:11:43.807147 11714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:43.807513 11714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:43.807627 11714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:43.815706 11714 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:43.815871 11714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:43.816004 11714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.68
I20240501 20:11:43.825702 11714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.68:33523
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.136.68
--webserver_port=38739
--tserver_master_addrs=127.28.136.126:40137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.68
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:43.827054 11714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:43.828909 11714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:43.841830 11722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:43.842175 11720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:43.842522 11719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:43.842927 11714 server_base.cc:1034] running on GCE node
I20240501 20:11:43.843576 11714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:43.844595 11714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:43.845799 11714 hybrid_clock.cc:648] HybridClock initialized: now 1714594303845732 us; error 65 us; skew 500 ppm
I20240501 20:11:43.846150 11714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:43.848539 11714 webserver.cc:460] Webserver started at http://127.28.136.68:38739/ using document root <none> and password file <none>
I20240501 20:11:43.849364 11714 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:43.849511 11714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:43.855175 11714 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:11:43.858292 11728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.860013 11714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:11:43.860208 11714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "add66495a78243b4af7fd39722c62782"
format_stamp: "Formatted at 2024-05-01 20:11:35 on dist-test-slave-sx0n"
I20240501 20:11:43.861423 11714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:43.879135 11714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:43.879981 11714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:43.880240 11714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:43.881263 11714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:43.882802 11714 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:43.882880 11714 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.882978 11714 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:43.883023 11714 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:43.912961 11714 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.68:33523
I20240501 20:11:43.912989 11840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.68:33523 every 8 connection(s)
I20240501 20:11:43.914480 11714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:43.919181 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 11714
I20240501 20:11:43.927662 11841 heartbeater.cc:347] Connected to a master server at 127.28.136.126:40137
I20240501 20:11:43.927994 11841 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:43.929006 11841 heartbeater.cc:511] Master 127.28.136.126:40137 requested a full tablet report, sending...
I20240501 20:11:43.935496 10813 ts_manager.cc:194] Re-registered known tserver with Master: add66495a78243b4af7fd39722c62782 (127.28.136.68:33523)
I20240501 20:11:43.936599 10813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.68:47635
I20240501 20:11:44.407639 11776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:44.417508 11067 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:44.428079 10936 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:44.430403 11645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:44.478549 11710 heartbeater.cc:503] Master 127.28.136.126:40137 was elected leader, sending a full tablet report...
I20240501 20:11:44.587189 11067 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:44.593330 11776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:44.598349 11645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:44.613245 10936 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:44.868175 11776 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:44.868148 11067 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:11:44.880594 10936 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:11:44.883023 11645 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
7fbcc621d8f34f9cbeaa2c20678149f3 | 127.28.136.126:40137 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.28.136.84:35131 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
220e903189f440beb7ac7a751dc00f26 | 127.28.136.66:46771 | HEALTHY | <none> | 6 | 0
add66495a78243b4af7fd39722c62782 | 127.28.136.68:33523 | HEALTHY | <none> | 0 | 0
ae8ee11327ae4e0b90c1b8c5e1bcb751 | 127.28.136.65:39213 | HEALTHY | <none> | 7 | 0
deb94eef62fc494fa1286ef07edb0f55 | 127.28.136.67:34987 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.28.136.65 | experimental | 127.28.136.65:39213
local_ip_for_outbound_sockets | 127.28.136.66 | experimental | 127.28.136.66:46771
local_ip_for_outbound_sockets | 127.28.136.67 | experimental | 127.28.136.67:34987
local_ip_for_outbound_sockets | 127.28.136.68 | experimental | 127.28.136.68:33523
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.28.136.65:39213
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.28.136.66:46771
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.28.136.67:34987
server_dump_info_path | /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.28.136.68:33523
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.28.136.84:35131 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:11:44.939287 29217 log_verifier.cc:126] Checking tablet 40d8a175340b49959801789ad7049ef2
I20240501 20:11:44.939605 11841 heartbeater.cc:503] Master 127.28.136.126:40137 was elected leader, sending a full tablet report...
I20240501 20:11:44.947762 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 40d8a175340b49959801789ad7049ef2
I20240501 20:11:44.947865 29217 log_verifier.cc:126] Checking tablet 5260ffd7948f41099f4e9017dd7a5fd0
I20240501 20:11:44.955904 29217 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 5260ffd7948f41099f4e9017dd7a5fd0
I20240501 20:11:44.956053 29217 log_verifier.cc:126] Checking tablet 5fce0bc3a7b2433a9fc82cbdf389a2b8
I20240501 20:11:44.963872 29217 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 5fce0bc3a7b2433a9fc82cbdf389a2b8
I20240501 20:11:44.963982 29217 log_verifier.cc:126] Checking tablet 60930f851c3146c980f69309684dd9c9
I20240501 20:11:44.971875 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 60930f851c3146c980f69309684dd9c9
I20240501 20:11:44.971980 29217 log_verifier.cc:126] Checking tablet 70a872401e3541ed81dfbb47b52877cd
I20240501 20:11:44.979629 29217 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 70a872401e3541ed81dfbb47b52877cd
I20240501 20:11:44.979728 29217 log_verifier.cc:126] Checking tablet 76c984ada15047fe9110dc8ef4bedae6
I20240501 20:11:44.986868 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 76c984ada15047fe9110dc8ef4bedae6
I20240501 20:11:44.986970 29217 log_verifier.cc:126] Checking tablet 80caff75bad24692882671275b46c6ff
I20240501 20:11:44.994108 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 80caff75bad24692882671275b46c6ff
I20240501 20:11:44.994199 29217 log_verifier.cc:126] Checking tablet 93e20193673049e3a1242470915fa790
I20240501 20:11:45.001405 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 93e20193673049e3a1242470915fa790
I20240501 20:11:45.001493 29217 log_verifier.cc:126] Checking tablet b3cc1b4bbd874be090c8eb9e2f2a0f65
I20240501 20:11:45.008654 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b3cc1b4bbd874be090c8eb9e2f2a0f65
I20240501 20:11:45.008770 29217 log_verifier.cc:126] Checking tablet c4a6bd894ba64689bb7c0fa5c5a48d9b
I20240501 20:11:45.016065 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c4a6bd894ba64689bb7c0fa5c5a48d9b
I20240501 20:11:45.016158 29217 log_verifier.cc:126] Checking tablet d44e00df30a04719bb7015cb54326eb5
I20240501 20:11:45.023766 29217 log_verifier.cc:177] Verified matching terms for 2 ops in tablet d44e00df30a04719bb7015cb54326eb5
I20240501 20:11:45.023869 29217 log_verifier.cc:126] Checking tablet dcc24131a5e047519a6cab0a96f3b914
I20240501 20:11:45.031273 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet dcc24131a5e047519a6cab0a96f3b914
I20240501 20:11:45.031365 29217 log_verifier.cc:126] Checking tablet fcd6b4412cad4573b5915989019ee0cc
I20240501 20:11:45.038710 29217 log_verifier.cc:177] Verified matching terms for 1 ops in tablet fcd6b4412cad4573b5915989019ee0cc
I20240501 20:11:45.039584 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 10874
I20240501 20:11:45.040972 10996 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:45.141047 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 10874
I20240501 20:11:45.158048 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 11005
I20240501 20:11:45.159078 11127 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:45.257910 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 11005
I20240501 20:11:45.274552 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 11583
I20240501 20:11:45.280447 11704 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:45.367602 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 11583
I20240501 20:11:45.382450 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 11714
I20240501 20:11:45.392830 11836 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:45.479725 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 11714
I20240501 20:11:45.494750 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 10784
I20240501 20:11:45.495855 10845 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:45.613021 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 10784
2024-05-01T20:11:45Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (12411 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (12411 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240501 20:11:45.662981 29217 test_util.cc:268] Using random seed: 411426797
I20240501 20:11:45.664788 29217 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:11:45.664853 29217 ts_itest-base.cc:113] --------------
I20240501 20:11:45.664896 29217 ts_itest-base.cc:114] 4 tablet servers
I20240501 20:11:45.664933 29217 ts_itest-base.cc:115] 2 replicas per TS
I20240501 20:11:45.664978 29217 ts_itest-base.cc:116] --------------
2024-05-01T20:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:11:45Z Disabled control of system clock
I20240501 20:11:45.726672 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:36673
--webserver_interface=127.28.136.126
--webserver_port=0
--builtin_ntp_servers=127.28.136.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.136.126:36673
--location_mapping_cmd=/tmp/dist-test-taskzuvgUY/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 20:11:46.066128 11956 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:11:46.066488 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:46.066596 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:46.066692 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:46.074178 11956 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:46.074306 11956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:11:46.074410 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:46.074463 11956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:11:46.074544 11956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:11:46.083994 11956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskzuvgUY/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.28.136.126:36673
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:36673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.136.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:46.085422 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:46.087152 11956 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:46.097370 11961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:46.097685 11964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:46.098240 11962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:46.098703 11956 server_base.cc:1034] running on GCE node
I20240501 20:11:46.099400 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:46.100714 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:46.101956 11956 hybrid_clock.cc:648] HybridClock initialized: now 1714594306101891 us; error 41 us; skew 500 ppm
I20240501 20:11:46.102263 11956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:46.104400 11956 webserver.cc:460] Webserver started at http://127.28.136.126:46859/ using document root <none> and password file <none>
I20240501 20:11:46.105113 11956 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:46.105239 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:46.105557 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:46.108084 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2537c03180b64034b6ecde0cdb4618cf"
format_stamp: "Formatted at 2024-05-01 20:11:46 on dist-test-slave-sx0n"
I20240501 20:11:46.108829 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2537c03180b64034b6ecde0cdb4618cf"
format_stamp: "Formatted at 2024-05-01 20:11:46 on dist-test-slave-sx0n"
I20240501 20:11:46.113571 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240501 20:11:46.116667 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:46.117949 11956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:11:46.118104 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2537c03180b64034b6ecde0cdb4618cf"
format_stamp: "Formatted at 2024-05-01 20:11:46 on dist-test-slave-sx0n"
I20240501 20:11:46.118276 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:46.136003 11956 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:46.136858 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:46.137118 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:46.153493 11956 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.126:36673
I20240501 20:11:46.153517 12021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.126:36673 every 8 connection(s)
I20240501 20:11:46.154816 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:11:46.156834 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 11956
I20240501 20:11:46.157076 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:11:46.159183 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:46.172755 12022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf: Bootstrap starting.
I20240501 20:11:46.175635 12022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:46.176601 12022 log.cc:824] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:46.179064 12022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf: No bootstrap required, opened a new log
I20240501 20:11:46.185485 12022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2537c03180b64034b6ecde0cdb4618cf" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36673 } }
I20240501 20:11:46.185812 12022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:46.185889 12022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537c03180b64034b6ecde0cdb4618cf, State: Initialized, Role: FOLLOWER
I20240501 20:11:46.186405 12022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2537c03180b64034b6ecde0cdb4618cf" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36673 } }
I20240501 20:11:46.186595 12022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:11:46.186695 12022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:11:46.186836 12022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:46.189201 12022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2537c03180b64034b6ecde0cdb4618cf" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36673 } }
I20240501 20:11:46.189649 12022 leader_election.cc:302] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2537c03180b64034b6ecde0cdb4618cf; no voters:
I20240501 20:11:46.190223 12022 leader_election.cc:288] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:11:46.190399 12027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:46.191009 12027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [term 1 LEADER]: Becoming Leader. State: Replica: 2537c03180b64034b6ecde0cdb4618cf, State: Running, Role: LEADER
I20240501 20:11:46.191541 12027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2537c03180b64034b6ecde0cdb4618cf" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36673 } }
I20240501 20:11:46.192096 12022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:11:46.194303 12029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2537c03180b64034b6ecde0cdb4618cf. Latest consensus state: current_term: 1 leader_uuid: "2537c03180b64034b6ecde0cdb4618cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2537c03180b64034b6ecde0cdb4618cf" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36673 } } }
I20240501 20:11:46.194674 12029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:46.194613 12028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2537c03180b64034b6ecde0cdb4618cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2537c03180b64034b6ecde0cdb4618cf" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 36673 } } }
I20240501 20:11:46.194890 12028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf [sys.catalog]: This master's current role is: LEADER
I20240501 20:11:46.196449 12033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:11:46.201927 12033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:11:46.210198 12033 catalog_manager.cc:1347] Generated new cluster ID: d15e778de66246ae8df8b80d0c7b3878
I20240501 20:11:46.210318 12033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:11:46.219760 12033 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:11:46.221151 12033 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:11:46.232302 12033 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2537c03180b64034b6ecde0cdb4618cf: Generated new TSK 0
I20240501 20:11:46.233239 12033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:11:46.245658 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.65:0
--local_ip_for_outbound_sockets=127.28.136.65
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36673
--builtin_ntp_servers=127.28.136.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:46.598918 12046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:46.599215 12046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:46.599351 12046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:46.607004 12046 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:46.607163 12046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:46.607326 12046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.65
I20240501 20:11:46.616665 12046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.65
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:46.618010 12046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:46.619719 12046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:46.632037 12051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:46.632480 12052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:46.632874 12054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:46.633337 12046 server_base.cc:1034] running on GCE node
I20240501 20:11:46.634006 12046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:46.634999 12046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:46.636222 12046 hybrid_clock.cc:648] HybridClock initialized: now 1714594306636162 us; error 50 us; skew 500 ppm
I20240501 20:11:46.636574 12046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:46.638967 12046 webserver.cc:460] Webserver started at http://127.28.136.65:34383/ using document root <none> and password file <none>
I20240501 20:11:46.639742 12046 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:46.639859 12046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:46.640221 12046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:46.642941 12046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f0843dc065a645b790a36bb1abd889ad"
format_stamp: "Formatted at 2024-05-01 20:11:46 on dist-test-slave-sx0n"
I20240501 20:11:46.643687 12046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f0843dc065a645b790a36bb1abd889ad"
format_stamp: "Formatted at 2024-05-01 20:11:46 on dist-test-slave-sx0n"
I20240501 20:11:46.648977 12046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20240501 20:11:46.652369 12060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:46.653813 12046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:46.653980 12046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f0843dc065a645b790a36bb1abd889ad"
format_stamp: "Formatted at 2024-05-01 20:11:46 on dist-test-slave-sx0n"
I20240501 20:11:46.654188 12046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:46.673493 12046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:46.674404 12046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:46.674690 12046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:46.675683 12046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:46.677340 12046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:46.677433 12046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:46.677557 12046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:46.677631 12046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:46.707708 12046 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.65:41511
I20240501 20:11:46.707757 12172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.65:41511 every 8 connection(s)
I20240501 20:11:46.709102 12046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:11:46.716921 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 12046
I20240501 20:11:46.717149 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:11:46.720604 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.66:0
--local_ip_for_outbound_sockets=127.28.136.66
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36673
--builtin_ntp_servers=127.28.136.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:46.723860 12173 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36673
I20240501 20:11:46.724160 12173 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:46.725061 12173 heartbeater.cc:511] Master 127.28.136.126:36673 requested a full tablet report, sending...
I20240501 20:11:46.794173 11987 ts_manager.cc:194] Registered new tserver with Master: f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:11:46.795934 11987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.65:39165
W20240501 20:11:47.067627 12178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:47.067960 12178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:47.068094 12178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:47.075575 12178 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:47.075722 12178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:47.075888 12178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.66
I20240501 20:11:47.085111 12178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.66
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:47.086467 12178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:47.088268 12178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:47.100555 12184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:47.100991 12183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:47.101364 12186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:47.102856 12178 server_base.cc:1034] running on GCE node
I20240501 20:11:47.103504 12178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:47.104513 12178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:47.105729 12178 hybrid_clock.cc:648] HybridClock initialized: now 1714594307105660 us; error 57 us; skew 500 ppm
I20240501 20:11:47.106060 12178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:47.108237 12178 webserver.cc:460] Webserver started at http://127.28.136.66:32819/ using document root <none> and password file <none>
I20240501 20:11:47.109031 12178 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:47.109160 12178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:47.109477 12178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:47.112030 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5e83f9a7e1da4d879f98535684683817"
format_stamp: "Formatted at 2024-05-01 20:11:47 on dist-test-slave-sx0n"
I20240501 20:11:47.112866 12178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5e83f9a7e1da4d879f98535684683817"
format_stamp: "Formatted at 2024-05-01 20:11:47 on dist-test-slave-sx0n"
I20240501 20:11:47.118007 12178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 20:11:47.121656 12192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:47.123031 12178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:11:47.123183 12178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5e83f9a7e1da4d879f98535684683817"
format_stamp: "Formatted at 2024-05-01 20:11:47 on dist-test-slave-sx0n"
I20240501 20:11:47.123373 12178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:47.142367 12178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:47.143239 12178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:47.143508 12178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:47.144491 12178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:47.146127 12178 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:47.146226 12178 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:47.146335 12178 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:47.146405 12178 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:47.181684 12178 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.66:36203
I20240501 20:11:47.181744 12304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.66:36203 every 8 connection(s)
I20240501 20:11:47.183005 12178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:11:47.190886 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 12178
I20240501 20:11:47.191082 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:11:47.194713 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:0
--local_ip_for_outbound_sockets=127.28.136.67
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36673
--builtin_ntp_servers=127.28.136.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:47.196442 12305 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36673
I20240501 20:11:47.196754 12305 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:47.197633 12305 heartbeater.cc:511] Master 127.28.136.126:36673 requested a full tablet report, sending...
I20240501 20:11:47.239873 11987 ts_manager.cc:194] Registered new tserver with Master: 5e83f9a7e1da4d879f98535684683817 (127.28.136.66:36203)
I20240501 20:11:47.241214 11987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.66:45473
W20240501 20:11:47.539605 12309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:47.539914 12309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:47.540001 12309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:47.547514 12309 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:47.547657 12309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:47.547784 12309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:11:47.557030 12309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:47.558377 12309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:47.560222 12309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:47.572901 12314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:47.573333 12317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:47.573678 12315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:47.574084 12309 server_base.cc:1034] running on GCE node
I20240501 20:11:47.574687 12309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:47.575656 12309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:47.576828 12309 hybrid_clock.cc:648] HybridClock initialized: now 1714594307576777 us; error 38 us; skew 500 ppm
I20240501 20:11:47.577157 12309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:47.579330 12309 webserver.cc:460] Webserver started at http://127.28.136.67:44485/ using document root <none> and password file <none>
I20240501 20:11:47.580116 12309 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:47.580225 12309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:47.580533 12309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:47.583164 12309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "91967a1a15304fdb8bb56ecf4501460d"
format_stamp: "Formatted at 2024-05-01 20:11:47 on dist-test-slave-sx0n"
I20240501 20:11:47.583940 12309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "91967a1a15304fdb8bb56ecf4501460d"
format_stamp: "Formatted at 2024-05-01 20:11:47 on dist-test-slave-sx0n"
I20240501 20:11:47.589023 12309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:11:47.592236 12323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:47.593644 12309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:47.593786 12309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "91967a1a15304fdb8bb56ecf4501460d"
format_stamp: "Formatted at 2024-05-01 20:11:47 on dist-test-slave-sx0n"
I20240501 20:11:47.593972 12309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:47.620275 12309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:47.621306 12309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:47.621569 12309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:47.622574 12309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:47.624058 12309 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:47.624146 12309 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:47.624256 12309 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:47.624321 12309 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:47.652400 12309 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:33283
I20240501 20:11:47.652467 12435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:33283 every 8 connection(s)
I20240501 20:11:47.653739 12309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:47.661442 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 12309
I20240501 20:11:47.661610 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:11:47.665210 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.68:0
--local_ip_for_outbound_sockets=127.28.136.68
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36673
--builtin_ntp_servers=127.28.136.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:47.666368 12436 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36673
I20240501 20:11:47.666687 12436 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:47.667670 12436 heartbeater.cc:511] Master 127.28.136.126:36673 requested a full tablet report, sending...
I20240501 20:11:47.713771 11987 ts_manager.cc:194] Registered new tserver with Master: 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67:33283)
I20240501 20:11:47.715081 11987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:37261
I20240501 20:11:47.799153 12173 heartbeater.cc:503] Master 127.28.136.126:36673 was elected leader, sending a full tablet report...
W20240501 20:11:48.006253 12440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:48.006541 12440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:48.006639 12440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:48.013986 12440 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:48.014097 12440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:48.014220 12440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.68
I20240501 20:11:48.023088 12440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:36673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.68
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:48.024329 12440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:48.026043 12440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:48.037765 12445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:48.038146 12446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:48.038385 12448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:48.038780 12440 server_base.cc:1034] running on GCE node
I20240501 20:11:48.039356 12440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:48.040282 12440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:48.041483 12440 hybrid_clock.cc:648] HybridClock initialized: now 1714594308041433 us; error 43 us; skew 500 ppm
I20240501 20:11:48.041800 12440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:48.043994 12440 webserver.cc:460] Webserver started at http://127.28.136.68:37949/ using document root <none> and password file <none>
I20240501 20:11:48.044756 12440 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:48.044881 12440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:48.045190 12440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:11:48.047828 12440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "734cac9320094c0b958aa626da00a329"
format_stamp: "Formatted at 2024-05-01 20:11:48 on dist-test-slave-sx0n"
I20240501 20:11:48.048590 12440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "734cac9320094c0b958aa626da00a329"
format_stamp: "Formatted at 2024-05-01 20:11:48 on dist-test-slave-sx0n"
I20240501 20:11:48.053543 12440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240501 20:11:48.056721 12454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:48.058037 12440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:11:48.058178 12440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "734cac9320094c0b958aa626da00a329"
format_stamp: "Formatted at 2024-05-01 20:11:48 on dist-test-slave-sx0n"
I20240501 20:11:48.058362 12440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:48.077311 12440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:48.078128 12440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:48.078397 12440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:48.079313 12440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:48.080801 12440 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:48.080895 12440 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:48.081007 12440 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:48.081076 12440 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:48.109338 12440 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.68:36659
I20240501 20:11:48.109417 12566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.68:36659 every 8 connection(s)
I20240501 20:11:48.110643 12440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:11:48.112635 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 12440
I20240501 20:11:48.112810 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:11:48.123093 12567 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36673
I20240501 20:11:48.123378 12567 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:48.124157 12567 heartbeater.cc:511] Master 127.28.136.126:36673 requested a full tablet report, sending...
I20240501 20:11:48.167033 11987 ts_manager.cc:194] Registered new tserver with Master: 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:48.168006 11987 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.68:44427
I20240501 20:11:48.170233 29217 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:11:48.186487 29217 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240501 20:11:48.186648 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 12309
I20240501 20:11:48.194229 12431 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:48.244555 12305 heartbeater.cc:503] Master 127.28.136.126:36673 was elected leader, sending a full tablet report...
I20240501 20:11:48.274173 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 12309
I20240501 20:11:48.288712 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 12178
I20240501 20:11:48.296826 12300 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:11:48.377269 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 12178
I20240501 20:11:49.171360 12567 heartbeater.cc:503] Master 127.28.136.126:36673 was elected leader, sending a full tablet report...
I20240501 20:11:52.145181 11986 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:11:52.151966 11986 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240501 20:11:52.182173 12107 tablet_service.cc:1467] Processing CreateTablet for tablet 899e2186657046f9b7a1640d73b4c6ae (DEFAULT_TABLE table=rebalance_test_table_0 [id=6805974dc6254581b857434a63570fb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:52.182224 12108 tablet_service.cc:1467] Processing CreateTablet for tablet 635cfee6291f429ebe0285ca0ad4f4a5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6805974dc6254581b857434a63570fb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:52.183701 12107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899e2186657046f9b7a1640d73b4c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.184151 12108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635cfee6291f429ebe0285ca0ad4f4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.183688 12501 tablet_service.cc:1467] Processing CreateTablet for tablet 899e2186657046f9b7a1640d73b4c6ae (DEFAULT_TABLE table=rebalance_test_table_0 [id=6805974dc6254581b857434a63570fb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:52.184473 12502 tablet_service.cc:1467] Processing CreateTablet for tablet 635cfee6291f429ebe0285ca0ad4f4a5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6805974dc6254581b857434a63570fb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:52.184765 12500 tablet_service.cc:1467] Processing CreateTablet for tablet e1b384c576bd49c18e49ad8aa8e9af46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6805974dc6254581b857434a63570fb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:52.185513 12106 tablet_service.cc:1467] Processing CreateTablet for tablet e1b384c576bd49c18e49ad8aa8e9af46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=6805974dc6254581b857434a63570fb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:52.185767 12501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899e2186657046f9b7a1640d73b4c6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.186034 12106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b384c576bd49c18e49ad8aa8e9af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.186164 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b384c576bd49c18e49ad8aa8e9af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.187104 12502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635cfee6291f429ebe0285ca0ad4f4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:52.197880 12589 tablet_bootstrap.cc:492] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:52.201318 12589 tablet_bootstrap.cc:654] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:52.201561 12590 tablet_bootstrap.cc:492] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:52.202649 12589 log.cc:824] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:52.205106 12590 tablet_bootstrap.cc:654] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:52.206301 12590 log.cc:824] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:52.206678 12589 tablet_bootstrap.cc:492] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:52.206974 12589 ts_tablet_manager.cc:1367] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240501 20:11:52.208909 12590 tablet_bootstrap.cc:492] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:52.209173 12590 ts_tablet_manager.cc:1367] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240501 20:11:52.215471 12589 raft_consensus.cc:348] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:52.215899 12589 raft_consensus.cc:374] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:52.216006 12589 raft_consensus.cc:729] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.216686 12589 consensus_queue.cc:260] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:52.217485 12590 raft_consensus.cc:348] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.217872 12590 raft_consensus.cc:374] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:52.217986 12590 raft_consensus.cc:729] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.218385 12589 ts_tablet_manager.cc:1398] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240501 20:11:52.218634 12590 consensus_queue.cc:260] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.218811 12589 tablet_bootstrap.cc:492] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:52.220163 12590 ts_tablet_manager.cc:1398] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240501 20:11:52.220400 12589 tablet_bootstrap.cc:654] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:52.220571 12590 tablet_bootstrap.cc:492] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:52.222270 12590 tablet_bootstrap.cc:654] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:52.223428 12589 tablet_bootstrap.cc:492] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:52.223565 12589 ts_tablet_manager.cc:1367] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 20:11:52.224423 12590 tablet_bootstrap.cc:492] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:52.224561 12590 ts_tablet_manager.cc:1367] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:52.224781 12589 raft_consensus.cc:348] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.224960 12589 raft_consensus.cc:374] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:52.225049 12589 raft_consensus.cc:729] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.225327 12589 consensus_queue.cc:260] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.225530 12590 raft_consensus.cc:348] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:52.225697 12590 raft_consensus.cc:374] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:52.225786 12590 raft_consensus.cc:729] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.225796 12589 ts_tablet_manager.cc:1398] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:52.226068 12589 tablet_bootstrap.cc:492] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:52.226012 12590 consensus_queue.cc:260] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:52.226497 12590 ts_tablet_manager.cc:1398] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:52.226805 12590 tablet_bootstrap.cc:492] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:52.227597 12589 tablet_bootstrap.cc:654] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:52.228302 12590 tablet_bootstrap.cc:654] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:52.230950 12589 tablet_bootstrap.cc:492] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:52.231068 12590 tablet_bootstrap.cc:492] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:52.231091 12589 ts_tablet_manager.cc:1367] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 20:11:52.231209 12590 ts_tablet_manager.cc:1367] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 20:11:52.231596 12589 raft_consensus.cc:348] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.231751 12589 raft_consensus.cc:374] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:52.231725 12590 raft_consensus.cc:348] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.231819 12589 raft_consensus.cc:729] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.231920 12590 raft_consensus.cc:374] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:52.232024 12590 raft_consensus.cc:729] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:52.232030 12589 consensus_queue.cc:260] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.232460 12589 ts_tablet_manager.cc:1398] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:52.232242 12590 consensus_queue.cc:260] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.233152 12590 ts_tablet_manager.cc:1398] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:11:52.233860 12593 raft_consensus.cc:482] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:52.234086 12593 raft_consensus.cc:504] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:52.235471 12593 leader_election.cc:288] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:52.241909 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635cfee6291f429ebe0285ca0ad4f4a5" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:52.242352 12522 raft_consensus.cc:2454] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 0.
I20240501 20:11:52.242867 12064 leader_election.cc:302] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:52.243201 12593 raft_consensus.cc:2781] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:52.243335 12593 raft_consensus.cc:482] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:52.243425 12593 raft_consensus.cc:3037] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:52.245415 12593 raft_consensus.cc:504] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:52.245890 12593 leader_election.cc:288] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:52.246326 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635cfee6291f429ebe0285ca0ad4f4a5" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:52.246505 12522 raft_consensus.cc:3037] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:52.248538 12522 raft_consensus.cc:2454] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 1.
I20240501 20:11:52.248903 12064 leader_election.cc:302] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:52.249255 12593 raft_consensus.cc:2781] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:52.249639 12593 raft_consensus.cc:686] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Leader. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:11:52.250118 12593 consensus_queue.cc:237] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:52.254668 11987 catalog_manager.cc:5484] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f0843dc065a645b790a36bb1abd889ad (127.28.136.65). New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:52.305639 12593 raft_consensus.cc:482] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:52.305876 12593 raft_consensus.cc:504] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.306401 12593 leader_election.cc:288] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:52.307029 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899e2186657046f9b7a1640d73b4c6ae" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:52.307274 12522 raft_consensus.cc:2454] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 0.
I20240501 20:11:52.307652 12064 leader_election.cc:302] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:52.307933 12593 raft_consensus.cc:2781] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:52.308023 12593 raft_consensus.cc:482] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:52.308100 12593 raft_consensus.cc:3037] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:52.309967 12593 raft_consensus.cc:504] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.310434 12593 leader_election.cc:288] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:52.310907 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899e2186657046f9b7a1640d73b4c6ae" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:52.311101 12522 raft_consensus.cc:3037] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:52.312978 12522 raft_consensus.cc:2454] T 899e2186657046f9b7a1640d73b4c6ae P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 1.
I20240501 20:11:52.313344 12064 leader_election.cc:302] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:52.313592 12593 raft_consensus.cc:2781] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:52.313750 12593 raft_consensus.cc:686] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Leader. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:11:52.313995 12593 consensus_queue.cc:237] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.316285 11987 catalog_manager.cc:5484] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f0843dc065a645b790a36bb1abd889ad (127.28.136.65). New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:52.359907 12594 raft_consensus.cc:482] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:52.360113 12594 raft_consensus.cc:504] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.361603 12594 leader_election.cc:288] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
W20240501 20:11:52.365155 12568 tablet.cc:2369] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:52.368129 12128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b384c576bd49c18e49ad8aa8e9af46" candidate_uuid: "734cac9320094c0b958aa626da00a329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0843dc065a645b790a36bb1abd889ad" is_pre_election: true
I20240501 20:11:52.368520 12128 raft_consensus.cc:2454] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 734cac9320094c0b958aa626da00a329 in term 0.
I20240501 20:11:52.369060 12455 leader_election.cc:302] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:52.369415 12594 raft_consensus.cc:2781] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:52.369539 12594 raft_consensus.cc:482] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:52.369626 12594 raft_consensus.cc:3037] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:52.371562 12594 raft_consensus.cc:504] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.372042 12594 leader_election.cc:288] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [CANDIDATE]: Term 1 election: Requested vote from peers f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:11:52.372546 12128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b384c576bd49c18e49ad8aa8e9af46" candidate_uuid: "734cac9320094c0b958aa626da00a329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
I20240501 20:11:52.372728 12128 raft_consensus.cc:3037] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:52.374934 12128 raft_consensus.cc:2454] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 734cac9320094c0b958aa626da00a329 in term 1.
I20240501 20:11:52.375329 12455 leader_election.cc:302] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:52.375604 12594 raft_consensus.cc:2781] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:52.376111 12594 raft_consensus.cc:686] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 1 LEADER]: Becoming Leader. State: Replica: 734cac9320094c0b958aa626da00a329, State: Running, Role: LEADER
I20240501 20:11:52.376556 12594 consensus_queue.cc:237] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:52.381147 11987 catalog_manager.cc:5484] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 734cac9320094c0b958aa626da00a329 (127.28.136.68). New cstate: current_term: 1 leader_uuid: "734cac9320094c0b958aa626da00a329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: UNKNOWN } } }
W20240501 20:11:52.464377 12174 tablet.cc:2369] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:11:52.674022 12593 consensus_queue.cc:1035] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:52.726400 12593 consensus_queue.cc:1035] T 899e2186657046f9b7a1640d73b4c6ae P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:52.751070 12606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:52.751416 12606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:52.759076 12606 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:52.762766 12594 consensus_queue.cc:1035] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 144.08 ms
time per row: 0.0720402 ms
I20240501 20:11:53.047533 11987 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:11:53.048324 11987 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240501 20:11:53.056224 12107 tablet_service.cc:1467] Processing CreateTablet for tablet 8b31993c6ba74104a7382494ebcf3244 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a254aee70d64baa8a86ed14cd11bf6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:53.056304 12502 tablet_service.cc:1467] Processing CreateTablet for tablet 8b31993c6ba74104a7382494ebcf3244 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a254aee70d64baa8a86ed14cd11bf6e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:53.056753 12502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b31993c6ba74104a7382494ebcf3244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.056762 12107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b31993c6ba74104a7382494ebcf3244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.057250 12501 tablet_service.cc:1467] Processing CreateTablet for tablet dffffa1de8d248aaa6d2bcb2621eb5a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a254aee70d64baa8a86ed14cd11bf6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:53.057675 12501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffffa1de8d248aaa6d2bcb2621eb5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.058079 12106 tablet_service.cc:1467] Processing CreateTablet for tablet dffffa1de8d248aaa6d2bcb2621eb5a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a254aee70d64baa8a86ed14cd11bf6e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:53.058527 12106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffffa1de8d248aaa6d2bcb2621eb5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.061110 12108 tablet_service.cc:1467] Processing CreateTablet for tablet 7bee467eecb94311ac5494db9ff9d640 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a254aee70d64baa8a86ed14cd11bf6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:53.061494 12108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bee467eecb94311ac5494db9ff9d640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.057684 12500 tablet_service.cc:1467] Processing CreateTablet for tablet 7bee467eecb94311ac5494db9ff9d640 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a254aee70d64baa8a86ed14cd11bf6e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:53.062398 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bee467eecb94311ac5494db9ff9d640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:53.071112 12658 tablet_bootstrap.cc:492] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:53.071460 12659 tablet_bootstrap.cc:492] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:53.073087 12658 tablet_bootstrap.cc:654] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:53.073247 12659 tablet_bootstrap.cc:654] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:53.075196 12658 tablet_bootstrap.cc:492] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:53.075295 12659 tablet_bootstrap.cc:492] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:53.075316 12658 ts_tablet_manager.cc:1367] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:53.075426 12659 ts_tablet_manager.cc:1367] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:53.075946 12658 raft_consensus.cc:348] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.076107 12658 raft_consensus.cc:374] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:53.076028 12659 raft_consensus.cc:348] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.076197 12659 raft_consensus.cc:374] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:53.076192 12658 raft_consensus.cc:729] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.076280 12659 raft_consensus.cc:729] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.076401 12658 consensus_queue.cc:260] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.076488 12659 consensus_queue.cc:260] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.076927 12658 ts_tablet_manager.cc:1398] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:53.077075 12659 ts_tablet_manager.cc:1398] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:53.077172 12658 tablet_bootstrap.cc:492] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:53.077344 12659 tablet_bootstrap.cc:492] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:53.078641 12658 tablet_bootstrap.cc:654] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:53.078827 12659 tablet_bootstrap.cc:654] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:53.081758 12659 tablet_bootstrap.cc:492] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:53.081909 12659 ts_tablet_manager.cc:1367] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 20:11:53.082054 12658 tablet_bootstrap.cc:492] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:53.082224 12658 ts_tablet_manager.cc:1367] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:11:53.082574 12659 raft_consensus.cc:348] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:53.082736 12659 raft_consensus.cc:374] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:53.082809 12659 raft_consensus.cc:729] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.082945 12658 raft_consensus.cc:348] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:53.083099 12658 raft_consensus.cc:374] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:53.083168 12658 raft_consensus.cc:729] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.083099 12659 consensus_queue.cc:260] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:53.083415 12658 consensus_queue.cc:260] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:53.083583 12659 ts_tablet_manager.cc:1398] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:53.083840 12658 ts_tablet_manager.cc:1398] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:53.083895 12659 tablet_bootstrap.cc:492] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:53.084183 12658 tablet_bootstrap.cc:492] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:53.085440 12659 tablet_bootstrap.cc:654] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:53.085882 12658 tablet_bootstrap.cc:654] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:53.089274 12659 tablet_bootstrap.cc:492] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:53.089380 12658 tablet_bootstrap.cc:492] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:53.089406 12659 ts_tablet_manager.cc:1367] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 20:11:53.089507 12658 ts_tablet_manager.cc:1367] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:11:53.089959 12659 raft_consensus.cc:348] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.090040 12658 raft_consensus.cc:348] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.090112 12659 raft_consensus.cc:374] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:53.090193 12658 raft_consensus.cc:374] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:53.090214 12659 raft_consensus.cc:729] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.090277 12658 raft_consensus.cc:729] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:53.090415 12659 consensus_queue.cc:260] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.090492 12658 consensus_queue.cc:260] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.090987 12658 ts_tablet_manager.cc:1398] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:11:53.091315 12659 ts_tablet_manager.cc:1398] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:53.287693 12593 raft_consensus.cc:482] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:53.287914 12593 raft_consensus.cc:504] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.288558 12593 leader_election.cc:288] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:53.289440 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b31993c6ba74104a7382494ebcf3244" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:53.289651 12522 raft_consensus.cc:2454] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 0.
I20240501 20:11:53.290177 12064 leader_election.cc:302] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:53.290482 12593 raft_consensus.cc:2781] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:53.290587 12593 raft_consensus.cc:482] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:53.290650 12593 raft_consensus.cc:3037] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:53.292547 12593 raft_consensus.cc:504] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.293066 12593 leader_election.cc:288] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:53.293651 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b31993c6ba74104a7382494ebcf3244" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:53.293830 12522 raft_consensus.cc:3037] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:53.295693 12522 raft_consensus.cc:2454] T 8b31993c6ba74104a7382494ebcf3244 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 1.
I20240501 20:11:53.296073 12064 leader_election.cc:302] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:53.296365 12593 raft_consensus.cc:2781] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:53.296506 12593 raft_consensus.cc:686] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Leader. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:11:53.296814 12593 consensus_queue.cc:237] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.299472 11985 catalog_manager.cc:5484] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f0843dc065a645b790a36bb1abd889ad (127.28.136.65). New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:53.371304 12593 raft_consensus.cc:482] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:53.371515 12593 raft_consensus.cc:504] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.372020 12593 leader_election.cc:288] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:53.372550 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffffa1de8d248aaa6d2bcb2621eb5a5" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:53.372817 12522 raft_consensus.cc:2454] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 0.
I20240501 20:11:53.373247 12064 leader_election.cc:302] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:53.373571 12593 raft_consensus.cc:2781] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:53.373665 12593 raft_consensus.cc:482] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:53.373759 12593 raft_consensus.cc:3037] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:53.375593 12593 raft_consensus.cc:504] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.376067 12593 leader_election.cc:288] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:53.376586 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffffa1de8d248aaa6d2bcb2621eb5a5" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:53.376832 12522 raft_consensus.cc:3037] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:53.378676 12522 raft_consensus.cc:2454] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 1.
I20240501 20:11:53.379009 12064 leader_election.cc:302] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:53.379244 12593 raft_consensus.cc:2781] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:53.379391 12593 raft_consensus.cc:686] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Leader. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:11:53.379629 12593 consensus_queue.cc:237] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:53.381906 11985 catalog_manager.cc:5484] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f0843dc065a645b790a36bb1abd889ad (127.28.136.65). New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:53.405055 12593 raft_consensus.cc:482] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:53.405223 12593 raft_consensus.cc:504] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:53.405683 12593 leader_election.cc:288] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:53.406165 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bee467eecb94311ac5494db9ff9d640" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:53.406368 12522 raft_consensus.cc:2454] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 0.
I20240501 20:11:53.406734 12064 leader_election.cc:302] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:53.406997 12593 raft_consensus.cc:2781] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:53.407132 12593 raft_consensus.cc:482] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:53.407209 12593 raft_consensus.cc:3037] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:53.409044 12593 raft_consensus.cc:504] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:53.409502 12593 leader_election.cc:288] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:53.410041 12522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bee467eecb94311ac5494db9ff9d640" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:53.410209 12522 raft_consensus.cc:3037] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:53.412003 12522 raft_consensus.cc:2454] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 1.
I20240501 20:11:53.412355 12064 leader_election.cc:302] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:53.412631 12593 raft_consensus.cc:2781] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:53.412791 12593 raft_consensus.cc:686] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Leader. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:11:53.413045 12593 consensus_queue.cc:237] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:53.415354 11985 catalog_manager.cc:5484] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f0843dc065a645b790a36bb1abd889ad (127.28.136.65). New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:53.682269 12593 consensus_queue.cc:1035] T 8b31993c6ba74104a7382494ebcf3244 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:53.771179 12669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:53.771525 12669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:53.779114 12669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:11:53.868264 12593 consensus_queue.cc:1035] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:53.898775 12520 raft_consensus.cc:1261] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:11:53.899557 12593 consensus_queue.cc:1035] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:53.962996 12701 mvcc.cc:205] Tried to move back new op lower bound from 7022978309722284032 to 7022978307741319168. Current Snapshot: MvccSnapshot[applied={T|T < 7022978309722284032}]
INSERT report
rows total: 2000
time total: 160.249 ms
time per row: 0.0801245 ms
I20240501 20:11:54.079262 11984 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46920:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:11:54.080086 11984 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240501 20:11:54.088511 12106 tablet_service.cc:1467] Processing CreateTablet for tablet 73a887ce3b7d4bd39a808aa680b6d2be (DEFAULT_TABLE table=rebalance_test_table_2 [id=2a5273edb4714e139e323ba0d82638d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:54.088683 12501 tablet_service.cc:1467] Processing CreateTablet for tablet 73a887ce3b7d4bd39a808aa680b6d2be (DEFAULT_TABLE table=rebalance_test_table_2 [id=2a5273edb4714e139e323ba0d82638d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:54.089215 12501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a887ce3b7d4bd39a808aa680b6d2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.089264 12106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a887ce3b7d4bd39a808aa680b6d2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.089640 12107 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4eb36913e1465990207cee2f766bd2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2a5273edb4714e139e323ba0d82638d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:54.089814 12500 tablet_service.cc:1467] Processing CreateTablet for tablet ffa1feb0e6b7416d9a8635bac610e835 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2a5273edb4714e139e323ba0d82638d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:54.090009 12107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4eb36913e1465990207cee2f766bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.090178 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa1feb0e6b7416d9a8635bac610e835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.092156 12502 tablet_service.cc:1467] Processing CreateTablet for tablet 1d4eb36913e1465990207cee2f766bd2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2a5273edb4714e139e323ba0d82638d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:54.092559 12502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4eb36913e1465990207cee2f766bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.092689 12108 tablet_service.cc:1467] Processing CreateTablet for tablet ffa1feb0e6b7416d9a8635bac610e835 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2a5273edb4714e139e323ba0d82638d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:54.093047 12108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffa1feb0e6b7416d9a8635bac610e835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:54.102239 12720 tablet_bootstrap.cc:492] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:54.102543 12721 tablet_bootstrap.cc:492] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:54.104038 12720 tablet_bootstrap.cc:654] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:54.104279 12721 tablet_bootstrap.cc:654] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:54.105899 12720 tablet_bootstrap.cc:492] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:54.106105 12721 tablet_bootstrap.cc:492] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:54.106236 12721 ts_tablet_manager.cc:1367] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:11:54.106242 12720 ts_tablet_manager.cc:1367] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:11:54.106887 12721 raft_consensus.cc:348] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.107095 12721 raft_consensus.cc:374] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:54.107025 12720 raft_consensus.cc:348] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.107206 12721 raft_consensus.cc:729] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.107218 12720 raft_consensus.cc:374] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:54.107296 12720 raft_consensus.cc:729] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.107434 12721 consensus_queue.cc:260] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.107508 12720 consensus_queue.cc:260] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.107971 12721 ts_tablet_manager.cc:1398] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:54.107986 12720 ts_tablet_manager.cc:1398] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:54.108225 12720 tablet_bootstrap.cc:492] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:54.108543 12721 tablet_bootstrap.cc:492] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:54.110407 12721 tablet_bootstrap.cc:654] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:54.111414 12720 tablet_bootstrap.cc:654] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:54.114385 12721 tablet_bootstrap.cc:492] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:54.114576 12721 ts_tablet_manager.cc:1367] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240501 20:11:54.114946 12720 tablet_bootstrap.cc:492] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:54.115120 12720 ts_tablet_manager.cc:1367] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 20:11:54.115357 12721 raft_consensus.cc:348] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:54.115578 12721 raft_consensus.cc:374] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:54.115665 12721 raft_consensus.cc:729] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.115895 12720 raft_consensus.cc:348] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.115909 12721 consensus_queue.cc:260] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:54.116114 12720 raft_consensus.cc:374] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:54.116230 12720 raft_consensus.cc:729] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.116474 12721 ts_tablet_manager.cc:1398] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:11:54.116499 12720 consensus_queue.cc:260] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.116815 12721 tablet_bootstrap.cc:492] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: Bootstrap starting.
I20240501 20:11:54.118005 12720 ts_tablet_manager.cc:1398] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:11:54.118335 12720 tablet_bootstrap.cc:492] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: Bootstrap starting.
I20240501 20:11:54.118402 12721 tablet_bootstrap.cc:654] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:54.121803 12720 tablet_bootstrap.cc:654] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:54.122608 12721 tablet_bootstrap.cc:492] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: No bootstrap required, opened a new log
I20240501 20:11:54.122818 12721 ts_tablet_manager.cc:1367] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:11:54.123526 12721 raft_consensus.cc:348] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.123716 12720 tablet_bootstrap.cc:492] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: No bootstrap required, opened a new log
I20240501 20:11:54.123731 12721 raft_consensus.cc:374] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:54.123816 12721 raft_consensus.cc:729] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.123937 12720 ts_tablet_manager.cc:1367] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 20:11:54.124048 12721 consensus_queue.cc:260] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.124606 12721 ts_tablet_manager.cc:1398] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:54.124687 12720 raft_consensus.cc:348] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:54.124888 12720 raft_consensus.cc:374] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:54.124990 12720 raft_consensus.cc:729] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734cac9320094c0b958aa626da00a329, State: Initialized, Role: FOLLOWER
I20240501 20:11:54.125344 12720 consensus_queue.cc:260] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:54.125910 12720 ts_tablet_manager.cc:1398] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:54.393618 12708 raft_consensus.cc:482] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:54.393819 12708 raft_consensus.cc:504] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.394377 12708 leader_election.cc:288] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:54.394968 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a887ce3b7d4bd39a808aa680b6d2be" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:54.395190 12521 raft_consensus.cc:2454] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 0.
I20240501 20:11:54.395646 12064 leader_election.cc:302] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:54.395908 12708 raft_consensus.cc:2781] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:54.396021 12708 raft_consensus.cc:482] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:54.396104 12708 raft_consensus.cc:3037] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:54.398396 12708 raft_consensus.cc:504] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.398833 12708 leader_election.cc:288] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:54.399341 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a887ce3b7d4bd39a808aa680b6d2be" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:54.399515 12521 raft_consensus.cc:3037] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:54.401394 12521 raft_consensus.cc:2454] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 1.
I20240501 20:11:54.401777 12064 leader_election.cc:302] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:54.402048 12708 raft_consensus.cc:2781] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:54.402172 12708 raft_consensus.cc:686] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Leader. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:11:54.402421 12708 consensus_queue.cc:237] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.404616 11984 catalog_manager.cc:5484] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f0843dc065a645b790a36bb1abd889ad (127.28.136.65). New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:54.419256 12708 raft_consensus.cc:482] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:54.419487 12708 raft_consensus.cc:504] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:54.420078 12708 leader_election.cc:288] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:54.420672 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4eb36913e1465990207cee2f766bd2" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:54.420964 12521 raft_consensus.cc:2454] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 0.
I20240501 20:11:54.421360 12064 leader_election.cc:302] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:54.421658 12708 raft_consensus.cc:2781] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:54.421749 12708 raft_consensus.cc:482] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:54.421833 12708 raft_consensus.cc:3037] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:54.423683 12708 raft_consensus.cc:504] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:54.424165 12708 leader_election.cc:288] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:54.424688 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d4eb36913e1465990207cee2f766bd2" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:54.424916 12521 raft_consensus.cc:3037] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:54.426765 12521 raft_consensus.cc:2454] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 1.
I20240501 20:11:54.427160 12064 leader_election.cc:302] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:54.427419 12708 raft_consensus.cc:2781] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:54.427580 12708 raft_consensus.cc:686] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Leader. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:11:54.427829 12708 consensus_queue.cc:237] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } }
I20240501 20:11:54.430351 11984 catalog_manager.cc:5484] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad reported cstate change: term changed from 0 to 1, leader changed from <none> to f0843dc065a645b790a36bb1abd889ad (127.28.136.65). New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:54.531232 12708 raft_consensus.cc:482] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:54.531440 12708 raft_consensus.cc:504] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.531626 12594 raft_consensus.cc:482] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:54.531800 12594 raft_consensus.cc:504] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.532013 12708 leader_election.cc:288] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:54.532346 12594 leader_election.cc:288] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:11:54.532625 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa1feb0e6b7416d9a8635bac610e835" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:54.532866 12521 raft_consensus.cc:2454] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 0.
I20240501 20:11:54.532845 12128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa1feb0e6b7416d9a8635bac610e835" candidate_uuid: "734cac9320094c0b958aa626da00a329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0843dc065a645b790a36bb1abd889ad" is_pre_election: true
I20240501 20:11:54.533082 12128 raft_consensus.cc:2454] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 734cac9320094c0b958aa626da00a329 in term 0.
I20240501 20:11:54.533301 12064 leader_election.cc:302] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:54.533525 12455 leader_election.cc:302] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:54.533586 12708 raft_consensus.cc:2781] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:54.533700 12708 raft_consensus.cc:482] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:54.533762 12594 raft_consensus.cc:2781] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:54.533784 12708 raft_consensus.cc:3037] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:54.533874 12594 raft_consensus.cc:482] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:54.533946 12594 raft_consensus.cc:3037] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:54.535570 12708 raft_consensus.cc:504] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.535610 12594 raft_consensus.cc:504] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:54.535972 12594 leader_election.cc:288] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [CANDIDATE]: Term 1 election: Requested vote from peers f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:11:54.535980 12708 leader_election.cc:288] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:54.536427 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa1feb0e6b7416d9a8635bac610e835" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:54.536455 12128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa1feb0e6b7416d9a8635bac610e835" candidate_uuid: "734cac9320094c0b958aa626da00a329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
I20240501 20:11:54.536661 12521 raft_consensus.cc:2379] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f0843dc065a645b790a36bb1abd889ad in current term 1: Already voted for candidate 734cac9320094c0b958aa626da00a329 in this term.
I20240501 20:11:54.536684 12128 raft_consensus.cc:2379] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 734cac9320094c0b958aa626da00a329 in current term 1: Already voted for candidate f0843dc065a645b790a36bb1abd889ad in this term.
I20240501 20:11:54.537070 12455 leader_election.cc:302] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 734cac9320094c0b958aa626da00a329; no voters: f0843dc065a645b790a36bb1abd889ad
I20240501 20:11:54.537079 12064 leader_election.cc:302] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f0843dc065a645b790a36bb1abd889ad; no voters: 734cac9320094c0b958aa626da00a329
I20240501 20:11:54.537333 12594 raft_consensus.cc:2726] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:11:54.537370 12708 raft_consensus.cc:2726] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:11:54.873489 12708 consensus_queue.cc:1035] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:55.000236 12708 consensus_queue.cc:1035] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:56.219781 12708 raft_consensus.cc:482] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:56.220023 12708 raft_consensus.cc:504] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:56.220582 12708 leader_election.cc:288] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:56.221287 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa1feb0e6b7416d9a8635bac610e835" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329" is_pre_election: true
I20240501 20:11:56.221503 12521 raft_consensus.cc:2454] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 1.
I20240501 20:11:56.222021 12064 leader_election.cc:302] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:56.222337 12708 raft_consensus.cc:2781] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:11:56.222425 12708 raft_consensus.cc:482] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:56.222517 12708 raft_consensus.cc:3037] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:56.224850 12708 raft_consensus.cc:504] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:56.225337 12708 leader_election.cc:288] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 2 election: Requested vote from peers 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:56.225915 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffa1feb0e6b7416d9a8635bac610e835" candidate_uuid: "f0843dc065a645b790a36bb1abd889ad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:11:56.226126 12521 raft_consensus.cc:3037] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:56.228018 12521 raft_consensus.cc:2454] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0843dc065a645b790a36bb1abd889ad in term 2.
I20240501 20:11:56.228444 12064 leader_election.cc:302] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, f0843dc065a645b790a36bb1abd889ad; no voters:
I20240501 20:11:56.228711 12708 raft_consensus.cc:2781] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:56.229091 12708 raft_consensus.cc:686] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [term 2 LEADER]: Becoming Leader. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:11:56.229343 12708 consensus_queue.cc:237] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } }
I20240501 20:11:56.231707 11984 catalog_manager.cc:5484] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad reported cstate change: term changed from 0 to 2, leader changed from <none> to f0843dc065a645b790a36bb1abd889ad (127.28.136.65). New cstate: current_term: 2 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:56.639683 12521 raft_consensus.cc:1261] T ffa1feb0e6b7416d9a8635bac610e835 P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240501 20:11:56.640254 12708 consensus_queue.cc:1035] T ffa1feb0e6b7416d9a8635bac610e835 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:57.072144 12744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:57.072530 12744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:57.079979 12744 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 156.916 ms
time per row: 0.0784581 ms
I20240501 20:11:57.373859 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:33283
--local_ip_for_outbound_sockets=127.28.136.67
--tserver_master_addrs=127.28.136.126:36673
--webserver_port=44485
--webserver_interface=127.28.136.67
--builtin_ntp_servers=127.28.136.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:57.718838 12794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:57.719205 12794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:57.719339 12794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:57.731372 12794 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:57.731544 12794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:57.731725 12794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:11:57.742697 12794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:33283
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=44485
--tserver_master_addrs=127.28.136.126:36673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:57.744061 12794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:57.745888 12794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:57.758940 12799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:57.759179 12802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:57.759768 12800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:57.762445 12794 server_base.cc:1034] running on GCE node
I20240501 20:11:57.763140 12794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:57.764256 12794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:57.765460 12794 hybrid_clock.cc:648] HybridClock initialized: now 1714594317765416 us; error 36 us; skew 500 ppm
I20240501 20:11:57.765780 12794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:57.768039 12794 webserver.cc:460] Webserver started at http://127.28.136.67:44485/ using document root <none> and password file <none>
I20240501 20:11:57.768842 12794 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:57.768997 12794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:57.774273 12794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20240501 20:11:57.777091 12808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:57.778368 12794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:11:57.778527 12794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "91967a1a15304fdb8bb56ecf4501460d"
format_stamp: "Formatted at 2024-05-01 20:11:47 on dist-test-slave-sx0n"
I20240501 20:11:57.779328 12794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:57.802227 12794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:57.803045 12794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:57.803311 12794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:57.804198 12794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:57.805689 12794 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:57.805768 12794 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:57.805889 12794 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:57.805940 12794 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:57.841470 12794 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:33283
I20240501 20:11:57.841504 12920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:33283 every 8 connection(s)
I20240501 20:11:57.842892 12794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:11:57.848014 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 12794
I20240501 20:11:57.848619 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.66:36203
--local_ip_for_outbound_sockets=127.28.136.66
--tserver_master_addrs=127.28.136.126:36673
--webserver_port=32819
--webserver_interface=127.28.136.66
--builtin_ntp_servers=127.28.136.84:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:57.855759 12921 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36673
I20240501 20:11:57.856104 12921 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:57.857183 12921 heartbeater.cc:511] Master 127.28.136.126:36673 requested a full tablet report, sending...
I20240501 20:11:57.859014 11984 ts_manager.cc:194] Re-registered known tserver with Master: 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67:33283)
I20240501 20:11:57.859891 11984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:56961
W20240501 20:11:58.194072 12925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:58.194381 12925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:58.194515 12925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:58.202260 12925 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:58.202395 12925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:58.202544 12925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.66
I20240501 20:11:58.211740 12925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.66:36203
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.136.66
--webserver_port=32819
--tserver_master_addrs=127.28.136.126:36673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.66
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:58.213075 12925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:58.214782 12925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:58.226534 12931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:58.227008 12934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:58.227370 12932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:58.227839 12925 server_base.cc:1034] running on GCE node
I20240501 20:11:58.228413 12925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:58.229423 12925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:58.230620 12925 hybrid_clock.cc:648] HybridClock initialized: now 1714594318230569 us; error 43 us; skew 500 ppm
I20240501 20:11:58.230935 12925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:58.233191 12925 webserver.cc:460] Webserver started at http://127.28.136.66:32819/ using document root <none> and password file <none>
I20240501 20:11:58.233947 12925 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:58.234069 12925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:58.239398 12925 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 20:11:58.242187 12940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:58.243541 12925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:11:58.243700 12925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5e83f9a7e1da4d879f98535684683817"
format_stamp: "Formatted at 2024-05-01 20:11:47 on dist-test-slave-sx0n"
I20240501 20:11:58.244511 12925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:58.268797 12925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:58.269712 12925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:58.270000 12925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:58.270968 12925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:58.272610 12925 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:58.272687 12925 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:58.272832 12925 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:58.272882 12925 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:58.300508 12925 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.66:36203
I20240501 20:11:58.300546 13052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.66:36203 every 8 connection(s)
I20240501 20:11:58.301961 12925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:11:58.306878 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 12925
I20240501 20:11:58.314517 13053 heartbeater.cc:347] Connected to a master server at 127.28.136.126:36673
I20240501 20:11:58.314834 13053 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:58.315645 13053 heartbeater.cc:511] Master 127.28.136.126:36673 requested a full tablet report, sending...
I20240501 20:11:58.317093 11984 ts_manager.cc:194] Re-registered known tserver with Master: 5e83f9a7e1da4d879f98535684683817 (127.28.136.66:36203)
I20240501 20:11:58.317893 11984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.66:52893
I20240501 20:11:58.800350 12502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:58.802407 12108 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:58.803539 12856 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:58.803800 12988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:58.862881 12921 heartbeater.cc:503] Master 127.28.136.126:36673 was elected leader, sending a full tablet report...
I20240501 20:11:58.964305 12502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:58.964305 12988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:58.971099 12856 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:58.972292 12108 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:59.119179 12856 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.126335 12108 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:59.128235 12988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.138749 12502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:59.287971 12988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.288914 12502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:59.292435 12108 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:59.306243 12856 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:59.320529 13053 heartbeater.cc:503] Master 127.28.136.126:36673 was elected leader, sending a full tablet report...
I20240501 20:11:59.372820 12128 consensus_queue.cc:237] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:11:59.375866 12522 raft_consensus.cc:1261] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:59.376485 13195 consensus_queue.cc:1035] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:59.379592 12788 raft_consensus.cc:2932] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } } }
I20240501 20:11:59.380486 12522 raft_consensus.cc:2932] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } } }
W20240501 20:11:59.382126 12064 consensus_peers.cc:485] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad -> Peer 5e83f9a7e1da4d879f98535684683817 (127.28.136.66:36203): Couldn't send request to peer 5e83f9a7e1da4d879f98535684683817. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d4eb36913e1465990207cee2f766bd2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:59.385908 11984 catalog_manager.cc:5484] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:59.398322 12128 consensus_queue.cc:237] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:11:59.400565 12522 raft_consensus.cc:1261] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:59.401170 13198 consensus_queue.cc:1035] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:59.403820 12788 raft_consensus.cc:2932] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } } }
I20240501 20:11:59.405815 12522 raft_consensus.cc:2932] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } } }
W20240501 20:11:59.406999 12062 consensus_peers.cc:485] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad -> Peer 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67:33283): Couldn't send request to peer 91967a1a15304fdb8bb56ecf4501460d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dffffa1de8d248aaa6d2bcb2621eb5a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:59.408646 11986 catalog_manager.cc:5484] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:59.419806 12128 consensus_queue.cc:237] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:11:59.421849 12522 raft_consensus.cc:1261] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:59.421972 12064 consensus_peers.cc:485] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad -> Peer 5e83f9a7e1da4d879f98535684683817 (127.28.136.66:36203): Couldn't send request to peer 5e83f9a7e1da4d879f98535684683817. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 635cfee6291f429ebe0285ca0ad4f4a5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:59.422417 12788 consensus_queue.cc:1035] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:59.424958 13198 raft_consensus.cc:2932] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } } }
I20240501 20:11:59.425683 12522 raft_consensus.cc:2932] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } } }
I20240501 20:11:59.430446 11984 catalog_manager.cc:5484] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:59.441982 12522 consensus_queue.cc:237] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:11:59.444384 12128 raft_consensus.cc:1261] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Refusing update from remote peer 734cac9320094c0b958aa626da00a329: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:11:59.444988 13060 consensus_queue.cc:1035] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:59.448033 13223 raft_consensus.cc:2932] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } } }
I20240501 20:11:59.448801 12128 raft_consensus.cc:2932] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } } }
W20240501 20:11:59.451047 12456 consensus_peers.cc:485] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 -> Peer 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67:33283): Couldn't send request to peer 91967a1a15304fdb8bb56ecf4501460d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1b384c576bd49c18e49ad8aa8e9af46. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:59.454103 11986 catalog_manager.cc:5484] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 reported cstate change: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New cstate: current_term: 1 leader_uuid: "734cac9320094c0b958aa626da00a329" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:59.466126 12128 consensus_queue.cc:237] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:11:59.468156 12522 raft_consensus.cc:1261] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:59.468411 12064 consensus_peers.cc:485] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad -> Peer 5e83f9a7e1da4d879f98535684683817 (127.28.136.66:36203): Couldn't send request to peer 5e83f9a7e1da4d879f98535684683817. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bee467eecb94311ac5494db9ff9d640. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:59.468688 13195 consensus_queue.cc:1035] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:59.471283 12788 raft_consensus.cc:2932] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } } }
I20240501 20:11:59.471956 12522 raft_consensus.cc:2932] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } } }
I20240501 20:11:59.476879 11984 catalog_manager.cc:5484] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index -1 to 4, NON_VOTER 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) added. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:59.489642 12128 consensus_queue.cc:237] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:11:59.491799 12522 raft_consensus.cc:1261] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:11:59.491886 12062 consensus_peers.cc:485] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad -> Peer 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67:33283): Couldn't send request to peer 91967a1a15304fdb8bb56ecf4501460d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73a887ce3b7d4bd39a808aa680b6d2be. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:11:59.492354 13195 consensus_queue.cc:1035] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:59.494845 12788 raft_consensus.cc:2932] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } } }
I20240501 20:11:59.495404 12522 raft_consensus.cc:2932] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } } }
I20240501 20:11:59.499935 11986 catalog_manager.cc:5484] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index -1 to 4, NON_VOTER 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) added. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:59.841099 13300 ts_tablet_manager.cc:902] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: Initiating tablet copy from peer f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:11:59.843132 13300 tablet_copy_client.cc:308] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:41511
I20240501 20:11:59.850677 12148 tablet_copy_service.cc:140] P f0843dc065a645b790a36bb1abd889ad: Received BeginTabletCopySession request for tablet dffffa1de8d248aaa6d2bcb2621eb5a5 from peer 91967a1a15304fdb8bb56ecf4501460d ({username='slave'} at 127.28.136.67:37097)
I20240501 20:11:59.850958 12148 tablet_copy_service.cc:161] P f0843dc065a645b790a36bb1abd889ad: Beginning new tablet copy session on tablet dffffa1de8d248aaa6d2bcb2621eb5a5 from peer 91967a1a15304fdb8bb56ecf4501460d at {username='slave'} at 127.28.136.67:37097: session id = 91967a1a15304fdb8bb56ecf4501460d-dffffa1de8d248aaa6d2bcb2621eb5a5
I20240501 20:11:59.853792 12148 tablet_copy_source_session.cc:205] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:59.858186 13300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffffa1de8d248aaa6d2bcb2621eb5a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:59.868854 13300 tablet_copy_client.cc:783] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:59.869156 13300 tablet_copy_client.cc:647] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:59.871305 13300 tablet_copy_client.cc:523] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:59.874459 13300 tablet_bootstrap.cc:492] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap starting.
I20240501 20:11:59.883155 13300 log.cc:824] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:59.889062 13307 ts_tablet_manager.cc:902] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: Initiating tablet copy from peer f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:11:59.890745 13307 tablet_copy_client.cc:308] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:41511
I20240501 20:11:59.897069 12148 tablet_copy_service.cc:140] P f0843dc065a645b790a36bb1abd889ad: Received BeginTabletCopySession request for tablet 1d4eb36913e1465990207cee2f766bd2 from peer 5e83f9a7e1da4d879f98535684683817 ({username='slave'} at 127.28.136.66:36381)
I20240501 20:11:59.897279 12148 tablet_copy_service.cc:161] P f0843dc065a645b790a36bb1abd889ad: Beginning new tablet copy session on tablet 1d4eb36913e1465990207cee2f766bd2 from peer 5e83f9a7e1da4d879f98535684683817 at {username='slave'} at 127.28.136.66:36381: session id = 5e83f9a7e1da4d879f98535684683817-1d4eb36913e1465990207cee2f766bd2
I20240501 20:11:59.900228 12148 tablet_copy_source_session.cc:205] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:59.904230 13307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d4eb36913e1465990207cee2f766bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:59.914000 13307 tablet_copy_client.cc:783] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:59.914326 13307 tablet_copy_client.cc:647] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:59.915836 13300 tablet_bootstrap.cc:492] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:59.916448 13300 tablet_bootstrap.cc:492] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap complete.
I20240501 20:11:59.916571 13307 tablet_copy_client.cc:523] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:59.917016 13300 ts_tablet_manager.cc:1367] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: Time spent bootstrapping tablet: real 0.043s user 0.040s sys 0.004s
I20240501 20:11:59.917145 13313 ts_tablet_manager.cc:902] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: Initiating tablet copy from peer 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:11:59.921316 13307 tablet_bootstrap.cc:492] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: Bootstrap starting.
I20240501 20:11:59.921772 13313 tablet_copy_client.cc:308] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:36659
I20240501 20:11:59.922928 13300 raft_consensus.cc:348] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:11:59.923334 13300 raft_consensus.cc:729] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91967a1a15304fdb8bb56ecf4501460d, State: Initialized, Role: LEARNER
I20240501 20:11:59.923844 13300 consensus_queue.cc:260] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:11:59.925701 13300 ts_tablet_manager.cc:1398] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:11:59.926811 12148 tablet_copy_service.cc:342] P f0843dc065a645b790a36bb1abd889ad: Request end of tablet copy session 91967a1a15304fdb8bb56ecf4501460d-dffffa1de8d248aaa6d2bcb2621eb5a5 received from {username='slave'} at 127.28.136.67:37097
I20240501 20:11:59.926983 12148 tablet_copy_service.cc:433] P f0843dc065a645b790a36bb1abd889ad: ending tablet copy session 91967a1a15304fdb8bb56ecf4501460d-dffffa1de8d248aaa6d2bcb2621eb5a5 on tablet dffffa1de8d248aaa6d2bcb2621eb5a5 with peer 91967a1a15304fdb8bb56ecf4501460d
I20240501 20:11:59.929903 12542 tablet_copy_service.cc:140] P 734cac9320094c0b958aa626da00a329: Received BeginTabletCopySession request for tablet e1b384c576bd49c18e49ad8aa8e9af46 from peer 91967a1a15304fdb8bb56ecf4501460d ({username='slave'} at 127.28.136.67:60647)
I20240501 20:11:59.930161 12542 tablet_copy_service.cc:161] P 734cac9320094c0b958aa626da00a329: Beginning new tablet copy session on tablet e1b384c576bd49c18e49ad8aa8e9af46 from peer 91967a1a15304fdb8bb56ecf4501460d at {username='slave'} at 127.28.136.67:60647: session id = 91967a1a15304fdb8bb56ecf4501460d-e1b384c576bd49c18e49ad8aa8e9af46
I20240501 20:11:59.933887 12542 tablet_copy_source_session.cc:205] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:59.934933 13307 log.cc:824] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: Log is configured to *not* fsync() on all Append() calls
I20240501 20:11:59.935703 13313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b384c576bd49c18e49ad8aa8e9af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:59.940135 13313 tablet_copy_client.cc:783] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:59.940376 13313 tablet_copy_client.cc:647] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:59.942255 13313 tablet_copy_client.cc:523] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:59.944514 13313 tablet_bootstrap.cc:492] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap starting.
I20240501 20:11:59.967137 13321 ts_tablet_manager.cc:902] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: Initiating tablet copy from peer f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:11:59.971132 13307 tablet_bootstrap.cc:492] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:59.972038 13307 tablet_bootstrap.cc:492] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: Bootstrap complete.
I20240501 20:11:59.972167 13321 tablet_copy_client.cc:308] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:41511
I20240501 20:11:59.972569 13307 ts_tablet_manager.cc:1367] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: Time spent bootstrapping tablet: real 0.052s user 0.045s sys 0.004s
I20240501 20:11:59.973048 12148 tablet_copy_service.cc:140] P f0843dc065a645b790a36bb1abd889ad: Received BeginTabletCopySession request for tablet 635cfee6291f429ebe0285ca0ad4f4a5 from peer 5e83f9a7e1da4d879f98535684683817 ({username='slave'} at 127.28.136.66:36381)
I20240501 20:11:59.973223 12148 tablet_copy_service.cc:161] P f0843dc065a645b790a36bb1abd889ad: Beginning new tablet copy session on tablet 635cfee6291f429ebe0285ca0ad4f4a5 from peer 5e83f9a7e1da4d879f98535684683817 at {username='slave'} at 127.28.136.66:36381: session id = 5e83f9a7e1da4d879f98535684683817-635cfee6291f429ebe0285ca0ad4f4a5
I20240501 20:11:59.976096 12148 tablet_copy_source_session.cc:205] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:59.977272 13300 ts_tablet_manager.cc:902] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: Initiating tablet copy from peer f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:11:59.977624 13321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635cfee6291f429ebe0285ca0ad4f4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:59.978935 13307 raft_consensus.cc:348] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:11:59.979442 13307 raft_consensus.cc:729] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5e83f9a7e1da4d879f98535684683817, State: Initialized, Role: LEARNER
I20240501 20:11:59.980103 13307 consensus_queue.cc:260] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:11:59.982386 13313 tablet_bootstrap.cc:492] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:11:59.982947 13313 tablet_bootstrap.cc:492] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap complete.
I20240501 20:11:59.983366 13313 ts_tablet_manager.cc:1367] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: Time spent bootstrapping tablet: real 0.039s user 0.036s sys 0.005s
I20240501 20:11:59.984588 13313 raft_consensus.cc:348] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:11:59.984997 13313 raft_consensus.cc:729] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91967a1a15304fdb8bb56ecf4501460d, State: Initialized, Role: LEARNER
I20240501 20:11:59.985276 13313 consensus_queue.cc:260] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:11:59.985818 13321 tablet_copy_client.cc:783] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Starting download of 0 data blocks...
I20240501 20:11:59.985970 13313 ts_tablet_manager.cc:1398] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:59.986088 13321 tablet_copy_client.cc:647] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Starting download of 1 WAL segments...
I20240501 20:11:59.986975 12542 tablet_copy_service.cc:342] P 734cac9320094c0b958aa626da00a329: Request end of tablet copy session 91967a1a15304fdb8bb56ecf4501460d-e1b384c576bd49c18e49ad8aa8e9af46 received from {username='slave'} at 127.28.136.67:60647
I20240501 20:11:59.987192 12542 tablet_copy_service.cc:433] P 734cac9320094c0b958aa626da00a329: ending tablet copy session 91967a1a15304fdb8bb56ecf4501460d-e1b384c576bd49c18e49ad8aa8e9af46 on tablet e1b384c576bd49c18e49ad8aa8e9af46 with peer 91967a1a15304fdb8bb56ecf4501460d
I20240501 20:11:59.988703 13321 tablet_copy_client.cc:523] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:11:59.990533 13307 ts_tablet_manager.cc:1398] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817: Time spent starting tablet: real 0.018s user 0.007s sys 0.007s
I20240501 20:11:59.992295 13321 tablet_bootstrap.cc:492] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: Bootstrap starting.
I20240501 20:11:59.992250 12148 tablet_copy_service.cc:342] P f0843dc065a645b790a36bb1abd889ad: Request end of tablet copy session 5e83f9a7e1da4d879f98535684683817-1d4eb36913e1465990207cee2f766bd2 received from {username='slave'} at 127.28.136.66:36381
I20240501 20:11:59.992421 12148 tablet_copy_service.cc:433] P f0843dc065a645b790a36bb1abd889ad: ending tablet copy session 5e83f9a7e1da4d879f98535684683817-1d4eb36913e1465990207cee2f766bd2 on tablet 1d4eb36913e1465990207cee2f766bd2 with peer 5e83f9a7e1da4d879f98535684683817
I20240501 20:11:59.992846 13300 tablet_copy_client.cc:308] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:41511
I20240501 20:11:59.993609 12148 tablet_copy_service.cc:140] P f0843dc065a645b790a36bb1abd889ad: Received BeginTabletCopySession request for tablet 73a887ce3b7d4bd39a808aa680b6d2be from peer 91967a1a15304fdb8bb56ecf4501460d ({username='slave'} at 127.28.136.67:37097)
I20240501 20:11:59.993793 12148 tablet_copy_service.cc:161] P f0843dc065a645b790a36bb1abd889ad: Beginning new tablet copy session on tablet 73a887ce3b7d4bd39a808aa680b6d2be from peer 91967a1a15304fdb8bb56ecf4501460d at {username='slave'} at 127.28.136.67:37097: session id = 91967a1a15304fdb8bb56ecf4501460d-73a887ce3b7d4bd39a808aa680b6d2be
I20240501 20:11:59.996532 12148 tablet_copy_source_session.cc:205] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:11:59.997815 13300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a887ce3b7d4bd39a808aa680b6d2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:00.002509 13300 tablet_copy_client.cc:783] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:00.002764 13300 tablet_copy_client.cc:647] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:00.004566 13300 tablet_copy_client.cc:523] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:00.007081 13300 tablet_bootstrap.cc:492] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap starting.
I20240501 20:12:00.033601 13321 tablet_bootstrap.cc:492] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:00.034422 13321 tablet_bootstrap.cc:492] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: Bootstrap complete.
I20240501 20:12:00.034704 13321 ts_tablet_manager.cc:1367] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: Time spent bootstrapping tablet: real 0.043s user 0.041s sys 0.000s
I20240501 20:12:00.035738 13321 raft_consensus.cc:348] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:12:00.035959 13321 raft_consensus.cc:729] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5e83f9a7e1da4d879f98535684683817, State: Initialized, Role: LEARNER
I20240501 20:12:00.036175 13321 consensus_queue.cc:260] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:12:00.037758 13321 ts_tablet_manager.cc:1398] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:00.038594 12148 tablet_copy_service.cc:342] P f0843dc065a645b790a36bb1abd889ad: Request end of tablet copy session 5e83f9a7e1da4d879f98535684683817-635cfee6291f429ebe0285ca0ad4f4a5 received from {username='slave'} at 127.28.136.66:36381
I20240501 20:12:00.038724 12148 tablet_copy_service.cc:433] P f0843dc065a645b790a36bb1abd889ad: ending tablet copy session 5e83f9a7e1da4d879f98535684683817-635cfee6291f429ebe0285ca0ad4f4a5 on tablet 635cfee6291f429ebe0285ca0ad4f4a5 with peer 5e83f9a7e1da4d879f98535684683817
I20240501 20:12:00.043175 13321 ts_tablet_manager.cc:902] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: Initiating tablet copy from peer f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511)
I20240501 20:12:00.048952 13321 tablet_copy_client.cc:308] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:41511
I20240501 20:12:00.049773 12148 tablet_copy_service.cc:140] P f0843dc065a645b790a36bb1abd889ad: Received BeginTabletCopySession request for tablet 7bee467eecb94311ac5494db9ff9d640 from peer 5e83f9a7e1da4d879f98535684683817 ({username='slave'} at 127.28.136.66:36381)
I20240501 20:12:00.050015 12148 tablet_copy_service.cc:161] P f0843dc065a645b790a36bb1abd889ad: Beginning new tablet copy session on tablet 7bee467eecb94311ac5494db9ff9d640 from peer 5e83f9a7e1da4d879f98535684683817 at {username='slave'} at 127.28.136.66:36381: session id = 5e83f9a7e1da4d879f98535684683817-7bee467eecb94311ac5494db9ff9d640
I20240501 20:12:00.050426 13300 tablet_bootstrap.cc:492] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:00.050887 13300 tablet_bootstrap.cc:492] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: Bootstrap complete.
I20240501 20:12:00.051199 13300 ts_tablet_manager.cc:1367] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: Time spent bootstrapping tablet: real 0.044s user 0.039s sys 0.005s
I20240501 20:12:00.052019 13300 raft_consensus.cc:348] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:12:00.052227 13300 raft_consensus.cc:729] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91967a1a15304fdb8bb56ecf4501460d, State: Initialized, Role: LEARNER
I20240501 20:12:00.052421 13300 consensus_queue.cc:260] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: true } }
I20240501 20:12:00.052646 12148 tablet_copy_source_session.cc:205] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:00.052975 13300 ts_tablet_manager.cc:1398] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:00.054026 12147 tablet_copy_service.cc:342] P f0843dc065a645b790a36bb1abd889ad: Request end of tablet copy session 91967a1a15304fdb8bb56ecf4501460d-73a887ce3b7d4bd39a808aa680b6d2be received from {username='slave'} at 127.28.136.67:37097
I20240501 20:12:00.054128 12147 tablet_copy_service.cc:433] P f0843dc065a645b790a36bb1abd889ad: ending tablet copy session 91967a1a15304fdb8bb56ecf4501460d-73a887ce3b7d4bd39a808aa680b6d2be on tablet 73a887ce3b7d4bd39a808aa680b6d2be with peer 91967a1a15304fdb8bb56ecf4501460d
I20240501 20:12:00.054204 13321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bee467eecb94311ac5494db9ff9d640. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:00.059618 13321 tablet_copy_client.cc:783] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:00.059832 13321 tablet_copy_client.cc:647] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:00.061429 13321 tablet_copy_client.cc:523] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:00.064632 13321 tablet_bootstrap.cc:492] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: Bootstrap starting.
I20240501 20:12:00.099684 13321 tablet_bootstrap.cc:492] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:00.100091 13321 tablet_bootstrap.cc:492] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: Bootstrap complete.
I20240501 20:12:00.100329 13321 ts_tablet_manager.cc:1367] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: Time spent bootstrapping tablet: real 0.036s user 0.035s sys 0.000s
I20240501 20:12:00.101042 13321 raft_consensus.cc:348] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:12:00.101238 13321 raft_consensus.cc:729] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5e83f9a7e1da4d879f98535684683817, State: Initialized, Role: LEARNER
I20240501 20:12:00.101429 13321 consensus_queue.cc:260] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: true } }
I20240501 20:12:00.101927 13321 ts_tablet_manager.cc:1398] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 20:12:00.102679 12148 tablet_copy_service.cc:342] P f0843dc065a645b790a36bb1abd889ad: Request end of tablet copy session 5e83f9a7e1da4d879f98535684683817-7bee467eecb94311ac5494db9ff9d640 received from {username='slave'} at 127.28.136.66:36381
I20240501 20:12:00.102813 12148 tablet_copy_service.cc:433] P f0843dc065a645b790a36bb1abd889ad: ending tablet copy session 5e83f9a7e1da4d879f98535684683817-7bee467eecb94311ac5494db9ff9d640 on tablet 7bee467eecb94311ac5494db9ff9d640 with peer 5e83f9a7e1da4d879f98535684683817
I20240501 20:12:00.266940 13008 raft_consensus.cc:1203] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:12:00.333726 12876 raft_consensus.cc:1203] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:12:00.381198 12876 raft_consensus.cc:1203] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:12:00.499467 12876 raft_consensus.cc:1203] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:12:00.524338 13008 raft_consensus.cc:1203] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:12:00.571271 13008 raft_consensus.cc:1203] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:12:00.732913 13195 raft_consensus.cc:1050] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: attempting to promote NON_VOTER 91967a1a15304fdb8bb56ecf4501460d to VOTER
I20240501 20:12:00.733721 13195 consensus_queue.cc:237] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:00.735721 12522 raft_consensus.cc:1261] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:00.735777 12876 raft_consensus.cc:1261] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:00.736348 12788 consensus_queue.cc:1035] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:00.736655 13368 consensus_queue.cc:1035] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:00.740401 12788 raft_consensus.cc:2932] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:00.741346 12522 raft_consensus.cc:2932] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:00.741251 12876 raft_consensus.cc:2932] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:00.746286 11984 catalog_manager.cc:5484] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:00.828660 13368 raft_consensus.cc:1050] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad: attempting to promote NON_VOTER 5e83f9a7e1da4d879f98535684683817 to VOTER
I20240501 20:12:00.829537 13368 consensus_queue.cc:237] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } }
I20240501 20:12:00.832329 12522 raft_consensus.cc:1261] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:00.832476 13008 raft_consensus.cc:1261] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:00.833453 13195 consensus_queue.cc:1035] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:00.834040 13368 consensus_queue.cc:1035] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:00.837383 13195 raft_consensus.cc:2932] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:00.840041 13008 raft_consensus.cc:2932] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:00.841851 12522 raft_consensus.cc:2932] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:00.842722 11986 catalog_manager.cc:5484] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:00.848603 12128 consensus_queue.cc:237] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } }
I20240501 20:12:00.850339 13008 raft_consensus.cc:1261] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:00.850872 13368 consensus_queue.cc:1035] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:00.853165 13195 raft_consensus.cc:2932] T 1d4eb36913e1465990207cee2f766bd2 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:00.857154 13008 raft_consensus.cc:2932] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:00.862332 11971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d4eb36913e1465990207cee2f766bd2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:00.862908 11986 catalog_manager.cc:5484] T 1d4eb36913e1465990207cee2f766bd2 P 5e83f9a7e1da4d879f98535684683817 reported cstate change: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:00.867388 13368 raft_consensus.cc:1050] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: attempting to promote NON_VOTER 91967a1a15304fdb8bb56ecf4501460d to VOTER
I20240501 20:12:00.868305 12502 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d4eb36913e1465990207cee2f766bd2 with delete_type TABLET_DATA_TOMBSTONED (TS 734cac9320094c0b958aa626da00a329 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34648
I20240501 20:12:00.868268 13368 consensus_queue.cc:237] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:00.871068 13461 tablet_replica.cc:330] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: stopping tablet replica
I20240501 20:12:00.871731 13461 raft_consensus.cc:2229] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:00.871922 12876 raft_consensus.cc:1261] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:00.872467 13461 raft_consensus.cc:2258] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:00.872617 13195 consensus_queue.cc:1035] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:00.872500 12522 raft_consensus.cc:1261] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:00.874086 13368 consensus_queue.cc:1035] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:00.874610 13461 ts_tablet_manager.cc:1875] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:00.877123 13368 raft_consensus.cc:2932] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:00.881877 12876 raft_consensus.cc:2932] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:00.882642 13461 ts_tablet_manager.cc:1888] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:00.882831 13461 log.cc:1196] T 1d4eb36913e1465990207cee2f766bd2 P 734cac9320094c0b958aa626da00a329: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/1d4eb36913e1465990207cee2f766bd2
I20240501 20:12:00.883965 11974 catalog_manager.cc:4830] TS 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659): tablet 1d4eb36913e1465990207cee2f766bd2 (table rebalance_test_table_2 [id=2a5273edb4714e139e323ba0d82638d2]) successfully deleted
I20240501 20:12:00.884145 11986 catalog_manager.cc:5484] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:00.892704 12521 raft_consensus.cc:2932] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:00.915005 12128 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dffffa1de8d248aaa6d2bcb2621eb5a5"
dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55628
I20240501 20:12:00.915284 12128 raft_consensus.cc:595] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:00.931565 12128 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73a887ce3b7d4bd39a808aa680b6d2be"
dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55644
I20240501 20:12:00.931799 12128 raft_consensus.cc:595] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:00.974812 12128 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dffffa1de8d248aaa6d2bcb2621eb5a5"
dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55660
I20240501 20:12:00.975036 12128 raft_consensus.cc:595] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:01.044452 13400 raft_consensus.cc:1050] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329: attempting to promote NON_VOTER 91967a1a15304fdb8bb56ecf4501460d to VOTER
I20240501 20:12:01.045331 13400 consensus_queue.cc:237] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.047683 12876 raft_consensus.cc:1261] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d [term 1 LEARNER]: Refusing update from remote peer 734cac9320094c0b958aa626da00a329: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:01.047720 12128 raft_consensus.cc:1261] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Refusing update from remote peer 734cac9320094c0b958aa626da00a329: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:01.048247 13515 consensus_queue.cc:1035] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:01.048617 13400 consensus_queue.cc:1035] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:01.051800 13515 raft_consensus.cc:2932] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.052271 12788 raft_consensus.cc:1050] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad: attempting to promote NON_VOTER 5e83f9a7e1da4d879f98535684683817 to VOTER
I20240501 20:12:01.052820 13195 raft_consensus.cc:1050] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad: attempting to promote NON_VOTER 5e83f9a7e1da4d879f98535684683817 to VOTER
I20240501 20:12:01.053103 12788 consensus_queue.cc:237] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } }
I20240501 20:12:01.053546 13195 consensus_queue.cc:237] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } }
I20240501 20:12:01.053828 12876 raft_consensus.cc:2932] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.056763 12128 raft_consensus.cc:2932] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.059912 11984 catalog_manager.cc:5484] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d reported cstate change: config changed from index 4 to 5, 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "734cac9320094c0b958aa626da00a329" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.063877 13007 raft_consensus.cc:1261] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:01.066375 13368 consensus_queue.cc:1035] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:01.067848 12521 raft_consensus.cc:1261] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:01.068450 13368 consensus_queue.cc:1035] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:01.078402 13368 raft_consensus.cc:2932] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.079026 13008 raft_consensus.cc:2932] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.084611 11986 catalog_manager.cc:5484] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 reported cstate change: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.085677 12521 raft_consensus.cc:2932] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.087375 12522 consensus_queue.cc:237] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.089762 12876 raft_consensus.cc:1261] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Refusing update from remote peer 734cac9320094c0b958aa626da00a329: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:01.090365 13517 consensus_queue.cc:1035] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:01.092986 13522 raft_consensus.cc:2932] T e1b384c576bd49c18e49ad8aa8e9af46 P 734cac9320094c0b958aa626da00a329 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.093652 12876 raft_consensus.cc:2932] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.097429 11974 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e1b384c576bd49c18e49ad8aa8e9af46 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:01.097736 13008 raft_consensus.cc:1261] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817 [term 1 LEARNER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:01.098284 12521 raft_consensus.cc:1261] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:01.099411 13368 consensus_queue.cc:1035] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:01.099651 11986 catalog_manager.cc:5484] T e1b384c576bd49c18e49ad8aa8e9af46 P 91967a1a15304fdb8bb56ecf4501460d reported cstate change: config changed from index 5 to 6, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New cstate: current_term: 1 leader_uuid: "734cac9320094c0b958aa626da00a329" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.099983 13368 consensus_queue.cc:1035] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:01.103785 12108 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b384c576bd49c18e49ad8aa8e9af46 with delete_type TABLET_DATA_TOMBSTONED (TS f0843dc065a645b790a36bb1abd889ad not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56012
I20240501 20:12:01.104671 12128 consensus_queue.cc:237] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } }
I20240501 20:12:01.109323 13531 tablet_replica.cc:330] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: stopping tablet replica
I20240501 20:12:01.109884 13531 raft_consensus.cc:2229] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:01.109941 13008 raft_consensus.cc:1261] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:01.110599 13531 raft_consensus.cc:2258] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:01.110515 13368 consensus_queue.cc:1035] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:01.113296 13008 raft_consensus.cc:2932] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.114415 13531 ts_tablet_manager.cc:1875] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:01.119143 11986 catalog_manager.cc:5484] T 635cfee6291f429ebe0285ca0ad4f4a5 P 5e83f9a7e1da4d879f98535684683817 reported cstate change: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.112723 13530 raft_consensus.cc:2932] T 635cfee6291f429ebe0285ca0ad4f4a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.122256 13529 raft_consensus.cc:2932] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.123049 12521 raft_consensus.cc:2932] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.124938 13531 ts_tablet_manager.cc:1888] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:01.125125 13531 log.cc:1196] T e1b384c576bd49c18e49ad8aa8e9af46 P f0843dc065a645b790a36bb1abd889ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/e1b384c576bd49c18e49ad8aa8e9af46
I20240501 20:12:01.125547 11971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 635cfee6291f429ebe0285ca0ad4f4a5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:01.126113 11971 catalog_manager.cc:4830] TS f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511): tablet e1b384c576bd49c18e49ad8aa8e9af46 (table rebalance_test_table_0 [id=6805974dc6254581b857434a63570fb2]) successfully deleted
I20240501 20:12:01.127408 12502 tablet_service.cc:1514] Processing DeleteTablet for tablet 635cfee6291f429ebe0285ca0ad4f4a5 with delete_type TABLET_DATA_TOMBSTONED (TS 734cac9320094c0b958aa626da00a329 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34648
I20240501 20:12:01.127796 13461 tablet_replica.cc:330] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: stopping tablet replica
I20240501 20:12:01.128026 13461 raft_consensus.cc:2229] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:01.128373 13461 raft_consensus.cc:2258] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:01.129417 13461 ts_tablet_manager.cc:1875] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:01.131652 11986 catalog_manager.cc:5484] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 reported cstate change: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.132881 13007 raft_consensus.cc:2932] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5e83f9a7e1da4d879f98535684683817 (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } attrs { replace: true } } peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.136016 13461 ts_tablet_manager.cc:1888] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:01.136148 13461 log.cc:1196] T 635cfee6291f429ebe0285ca0ad4f4a5 P 734cac9320094c0b958aa626da00a329: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/635cfee6291f429ebe0285ca0ad4f4a5
I20240501 20:12:01.136898 11974 catalog_manager.cc:4830] TS 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659): tablet 635cfee6291f429ebe0285ca0ad4f4a5 (table rebalance_test_table_0 [id=6805974dc6254581b857434a63570fb2]) successfully deleted
I20240501 20:12:01.142372 12128 consensus_queue.cc:237] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } }
I20240501 20:12:01.143949 13007 raft_consensus.cc:1261] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Refusing update from remote peer f0843dc065a645b790a36bb1abd889ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:01.144450 13368 consensus_queue.cc:1035] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:01.146507 13528 raft_consensus.cc:2932] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.147118 13007 raft_consensus.cc:2932] T 7bee467eecb94311ac5494db9ff9d640 P 5e83f9a7e1da4d879f98535684683817 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } } }
I20240501 20:12:01.150130 11971 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7bee467eecb94311ac5494db9ff9d640 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:01.151080 11983 catalog_manager.cc:5484] T 7bee467eecb94311ac5494db9ff9d640 P f0843dc065a645b790a36bb1abd889ad reported cstate change: config changed from index 5 to 6, VOTER 734cac9320094c0b958aa626da00a329 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "f0843dc065a645b790a36bb1abd889ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e83f9a7e1da4d879f98535684683817" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 36203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:01.154502 12502 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bee467eecb94311ac5494db9ff9d640 with delete_type TABLET_DATA_TOMBSTONED (TS 734cac9320094c0b958aa626da00a329 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34648
I20240501 20:12:01.154917 13461 tablet_replica.cc:330] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: stopping tablet replica
I20240501 20:12:01.155117 13461 raft_consensus.cc:2229] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:01.155437 13461 raft_consensus.cc:2258] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:01.156397 13461 ts_tablet_manager.cc:1875] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:01.162998 13461 ts_tablet_manager.cc:1888] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:01.163136 13461 log.cc:1196] T 7bee467eecb94311ac5494db9ff9d640 P 734cac9320094c0b958aa626da00a329: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/7bee467eecb94311ac5494db9ff9d640
I20240501 20:12:01.163933 11974 catalog_manager.cc:4830] TS 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659): tablet 7bee467eecb94311ac5494db9ff9d640 (table rebalance_test_table_1 [id=5a254aee70d64baa8a86ed14cd11bf6e]) successfully deleted
I20240501 20:12:01.188009 13528 raft_consensus.cc:979] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: : Instructing follower 91967a1a15304fdb8bb56ecf4501460d to start an election
I20240501 20:12:01.188230 13528 raft_consensus.cc:1067] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Signalling peer 91967a1a15304fdb8bb56ecf4501460d to start an election
I20240501 20:12:01.189211 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dffffa1de8d248aaa6d2bcb2621eb5a5"
dest_uuid: "91967a1a15304fdb8bb56ecf4501460d"
from {username='slave'} at 127.28.136.65:39601
I20240501 20:12:01.189548 12876 raft_consensus.cc:482] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:12:01.189742 12876 raft_consensus.cc:3037] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:01.192350 12876 raft_consensus.cc:504] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.192407 12128 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73a887ce3b7d4bd39a808aa680b6d2be"
dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55678
I20240501 20:12:01.192677 12128 raft_consensus.cc:595] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:01.194238 12876 leader_election.cc:288] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [CANDIDATE]: Term 2 election: Requested vote from peers f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511), 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:12:01.194547 12128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffffa1de8d248aaa6d2bcb2621eb5a5" candidate_uuid: "91967a1a15304fdb8bb56ecf4501460d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
I20240501 20:12:01.194777 12128 raft_consensus.cc:3032] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:12:01.194854 12128 raft_consensus.cc:729] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:12:01.194818 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffffa1de8d248aaa6d2bcb2621eb5a5" candidate_uuid: "91967a1a15304fdb8bb56ecf4501460d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:12:01.195061 12521 raft_consensus.cc:3037] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:01.195179 12128 consensus_queue.cc:260] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.195518 12128 raft_consensus.cc:3037] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:01.197199 12521 raft_consensus.cc:2454] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91967a1a15304fdb8bb56ecf4501460d in term 2.
I20240501 20:12:01.197481 12128 raft_consensus.cc:2454] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91967a1a15304fdb8bb56ecf4501460d in term 2.
I20240501 20:12:01.197803 12812 leader_election.cc:302] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, 91967a1a15304fdb8bb56ecf4501460d; no voters:
I20240501 20:12:01.198206 13318 raft_consensus.cc:2781] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:12:01.198818 13318 raft_consensus.cc:686] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 2 LEADER]: Becoming Leader. State: Replica: 91967a1a15304fdb8bb56ecf4501460d, State: Running, Role: LEADER
I20240501 20:12:01.199354 13318 consensus_queue.cc:237] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.203434 11986 catalog_manager.cc:5484] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d reported cstate change: term changed from 1 to 2, leader changed from f0843dc065a645b790a36bb1abd889ad (127.28.136.65) to 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67). New cstate: current_term: 2 leader_uuid: "91967a1a15304fdb8bb56ecf4501460d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:01.250656 12128 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73a887ce3b7d4bd39a808aa680b6d2be"
dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55710
I20240501 20:12:01.250890 12128 raft_consensus.cc:595] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:01.371660 13368 raft_consensus.cc:979] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: : Instructing follower 91967a1a15304fdb8bb56ecf4501460d to start an election
I20240501 20:12:01.371838 13368 raft_consensus.cc:1067] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Signalling peer 91967a1a15304fdb8bb56ecf4501460d to start an election
I20240501 20:12:01.372509 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73a887ce3b7d4bd39a808aa680b6d2be"
dest_uuid: "91967a1a15304fdb8bb56ecf4501460d"
from {username='slave'} at 127.28.136.65:39601
I20240501 20:12:01.372761 12876 raft_consensus.cc:482] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:12:01.372882 12876 raft_consensus.cc:3037] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:01.374876 12876 raft_consensus.cc:504] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.375500 12876 leader_election.cc:288] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [CANDIDATE]: Term 2 election: Requested vote from peers f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511), 734cac9320094c0b958aa626da00a329 (127.28.136.68:36659)
I20240501 20:12:01.375957 12128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a887ce3b7d4bd39a808aa680b6d2be" candidate_uuid: "91967a1a15304fdb8bb56ecf4501460d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f0843dc065a645b790a36bb1abd889ad"
I20240501 20:12:01.376030 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a887ce3b7d4bd39a808aa680b6d2be" candidate_uuid: "91967a1a15304fdb8bb56ecf4501460d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "734cac9320094c0b958aa626da00a329"
I20240501 20:12:01.376161 12128 raft_consensus.cc:3032] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:12:01.376248 12128 raft_consensus.cc:729] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f0843dc065a645b790a36bb1abd889ad, State: Running, Role: LEADER
I20240501 20:12:01.376256 12521 raft_consensus.cc:3037] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:01.376508 12128 consensus_queue.cc:260] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.376832 12128 raft_consensus.cc:3037] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:01.378229 12521 raft_consensus.cc:2454] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91967a1a15304fdb8bb56ecf4501460d in term 2.
I20240501 20:12:01.378672 12812 leader_election.cc:302] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 734cac9320094c0b958aa626da00a329, 91967a1a15304fdb8bb56ecf4501460d; no voters:
I20240501 20:12:01.378733 12128 raft_consensus.cc:2454] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91967a1a15304fdb8bb56ecf4501460d in term 2.
I20240501 20:12:01.379001 13318 raft_consensus.cc:2781] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:12:01.379158 13318 raft_consensus.cc:686] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 2 LEADER]: Becoming Leader. State: Replica: 91967a1a15304fdb8bb56ecf4501460d, State: Running, Role: LEADER
I20240501 20:12:01.379424 13318 consensus_queue.cc:237] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.381712 11986 catalog_manager.cc:5484] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d reported cstate change: term changed from 1 to 2, leader changed from f0843dc065a645b790a36bb1abd889ad (127.28.136.65) to 91967a1a15304fdb8bb56ecf4501460d (127.28.136.67). New cstate: current_term: 2 leader_uuid: "91967a1a15304fdb8bb56ecf4501460d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:01.773080 12521 raft_consensus.cc:1261] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Refusing update from remote peer 91967a1a15304fdb8bb56ecf4501460d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:01.773762 13318 consensus_queue.cc:1035] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:01.778326 12128 raft_consensus.cc:1261] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Refusing update from remote peer 91967a1a15304fdb8bb56ecf4501460d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:01.778959 13544 consensus_queue.cc:1035] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0843dc065a645b790a36bb1abd889ad" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:01.786588 12521 raft_consensus.cc:1261] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Refusing update from remote peer 91967a1a15304fdb8bb56ecf4501460d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:01.787182 13544 consensus_queue.cc:1035] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:01.790216 12875 consensus_queue.cc:237] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.792095 12521 raft_consensus.cc:1261] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Refusing update from remote peer 91967a1a15304fdb8bb56ecf4501460d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:12:01.793206 13544 consensus_queue.cc:1035] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:01.795569 12875 consensus_queue.cc:237] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } }
I20240501 20:12:01.795437 13318 raft_consensus.cc:2932] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.798550 11972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73a887ce3b7d4bd39a808aa680b6d2be with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:01.799299 12522 raft_consensus.cc:1261] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Refusing update from remote peer 91967a1a15304fdb8bb56ecf4501460d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:12:01.799407 12128 raft_consensus.cc:1261] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Refusing update from remote peer 91967a1a15304fdb8bb56ecf4501460d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:12:01.799170 12521 raft_consensus.cc:2932] T 73a887ce3b7d4bd39a808aa680b6d2be P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.799710 11983 catalog_manager.cc:5484] T 73a887ce3b7d4bd39a808aa680b6d2be P 91967a1a15304fdb8bb56ecf4501460d reported cstate change: config changed from index 5 to 7, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New cstate: current_term: 2 leader_uuid: "91967a1a15304fdb8bb56ecf4501460d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:01.799934 13318 consensus_queue.cc:1035] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [LEADER]: Connected to new peer: Peer: permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:01.803105 13544 raft_consensus.cc:2932] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.804081 12108 tablet_service.cc:1514] Processing DeleteTablet for tablet 73a887ce3b7d4bd39a808aa680b6d2be with delete_type TABLET_DATA_TOMBSTONED (TS f0843dc065a645b790a36bb1abd889ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56012
I20240501 20:12:01.806986 13608 tablet_replica.cc:330] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: stopping tablet replica
I20240501 20:12:01.807108 11972 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dffffa1de8d248aaa6d2bcb2621eb5a5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:01.807209 13608 raft_consensus.cc:2229] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:01.807416 13608 raft_consensus.cc:2258] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:01.808496 13608 ts_tablet_manager.cc:1875] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:01.808362 12521 raft_consensus.cc:2932] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 734cac9320094c0b958aa626da00a329 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } } }
I20240501 20:12:01.811408 11983 catalog_manager.cc:5484] T dffffa1de8d248aaa6d2bcb2621eb5a5 P 91967a1a15304fdb8bb56ecf4501460d reported cstate change: config changed from index 5 to 7, VOTER f0843dc065a645b790a36bb1abd889ad (127.28.136.65) evicted. New cstate: current_term: 2 leader_uuid: "91967a1a15304fdb8bb56ecf4501460d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "734cac9320094c0b958aa626da00a329" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 36659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "91967a1a15304fdb8bb56ecf4501460d" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 33283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:01.813977 13608 ts_tablet_manager.cc:1888] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:12:01.814121 13608 log.cc:1196] T 73a887ce3b7d4bd39a808aa680b6d2be P f0843dc065a645b790a36bb1abd889ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/73a887ce3b7d4bd39a808aa680b6d2be
I20240501 20:12:01.814994 12108 tablet_service.cc:1514] Processing DeleteTablet for tablet dffffa1de8d248aaa6d2bcb2621eb5a5 with delete_type TABLET_DATA_TOMBSTONED (TS f0843dc065a645b790a36bb1abd889ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56012
I20240501 20:12:01.815028 11971 catalog_manager.cc:4830] TS f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511): tablet 73a887ce3b7d4bd39a808aa680b6d2be (table rebalance_test_table_2 [id=2a5273edb4714e139e323ba0d82638d2]) successfully deleted
I20240501 20:12:01.815410 13608 tablet_replica.cc:330] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: stopping tablet replica
I20240501 20:12:01.815580 13608 raft_consensus.cc:2229] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:01.815742 13608 raft_consensus.cc:2258] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:01.816293 13608 ts_tablet_manager.cc:1875] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:01.820243 13608 ts_tablet_manager.cc:1888] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:12:01.820353 13608 log.cc:1196] T dffffa1de8d248aaa6d2bcb2621eb5a5 P f0843dc065a645b790a36bb1abd889ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/wals/dffffa1de8d248aaa6d2bcb2621eb5a5
I20240501 20:12:01.821018 11971 catalog_manager.cc:4830] TS f0843dc065a645b790a36bb1abd889ad (127.28.136.65:41511): tablet dffffa1de8d248aaa6d2bcb2621eb5a5 (table rebalance_test_table_1 [id=5a254aee70d64baa8a86ed14cd11bf6e]) successfully deleted
I20240501 20:12:02.002414 12856 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:02.005393 12988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.015070 12108 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:02.022572 12502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:02.191941 12502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:02.194103 12988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.204949 12856 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:02.208210 12108 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:02.342161 12988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:02.342952 12502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:02.346340 12856 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:12:02.349896 12108 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:12:02.530558 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 12046
I20240501 20:12:02.547502 12167 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:02.681214 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 12046
W20240501 20:12:02.703125 12941 connection.cc:535] server connection from 127.28.136.65:39401 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:12:02.703485 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 12925
I20240501 20:12:02.709232 13047 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:02.814131 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 12925
I20240501 20:12:02.831619 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 12794
I20240501 20:12:02.842216 12916 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:02.950994 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 12794
I20240501 20:12:02.969403 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 12440
I20240501 20:12:02.975203 12562 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:03.098572 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 12440
I20240501 20:12:03.117606 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 11956
I20240501 20:12:03.118800 12017 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:03.253602 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 11956
2024-05-01T20:12:03Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (17672 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240501 20:12:03.335834 29217 test_util.cc:268] Using random seed: 429099655
I20240501 20:12:03.337590 29217 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:12:03.337651 29217 ts_itest-base.cc:113] --------------
I20240501 20:12:03.337671 29217 ts_itest-base.cc:114] 10 tablet servers
I20240501 20:12:03.337728 29217 ts_itest-base.cc:115] 7 replicas per TS
I20240501 20:12:03.337777 29217 ts_itest-base.cc:116] --------------
2024-05-01T20:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:12:03Z Disabled control of system clock
I20240501 20:12:03.398207 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:34579
--webserver_interface=127.28.136.126
--webserver_port=0
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.136.126:34579
--location_mapping_cmd=/tmp/dist-test-taskzuvgUY/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240501 20:12:03.737625 13734 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:12:03.737993 13734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:03.738093 13734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:03.738195 13734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:03.745955 13734 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:03.746111 13734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:12:03.746217 13734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:03.746281 13734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:12:03.746369 13734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:12:03.756080 13734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskzuvgUY/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.28.136.126:34579
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.136.126:34579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.28.136.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:03.757532 13734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:03.759315 13734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:03.770467 13739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:03.770704 13742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:03.771136 13740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:03.771446 13734 server_base.cc:1034] running on GCE node
I20240501 20:12:03.772238 13734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:03.773782 13734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:03.774991 13734 hybrid_clock.cc:648] HybridClock initialized: now 1714594323774920 us; error 46 us; skew 500 ppm
I20240501 20:12:03.775313 13734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:03.777590 13734 webserver.cc:460] Webserver started at http://127.28.136.126:38861/ using document root <none> and password file <none>
I20240501 20:12:03.778303 13734 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:03.778429 13734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:03.778743 13734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:03.781436 13734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c"
format_stamp: "Formatted at 2024-05-01 20:12:03 on dist-test-slave-sx0n"
I20240501 20:12:03.782229 13734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c"
format_stamp: "Formatted at 2024-05-01 20:12:03 on dist-test-slave-sx0n"
I20240501 20:12:03.787354 13734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20240501 20:12:03.790779 13748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:03.792068 13734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:03.792238 13734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c"
format_stamp: "Formatted at 2024-05-01 20:12:03 on dist-test-slave-sx0n"
I20240501 20:12:03.792426 13734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:03.811055 13734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:03.811900 13734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:03.812199 13734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:03.829947 13734 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.126:34579
I20240501 20:12:03.829974 13799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.126:34579 every 8 connection(s)
I20240501 20:12:03.831465 13734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:12:03.835496 13800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:03.841128 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 13734
I20240501 20:12:03.841344 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:12:03.847067 13800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c: Bootstrap starting.
I20240501 20:12:03.850652 13800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:03.851732 13800 log.cc:824] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:03.855190 13800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c: No bootstrap required, opened a new log
I20240501 20:12:03.861644 13800 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 34579 } }
I20240501 20:12:03.862037 13800 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:03.862164 13800 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfefe0ff2f6496dba80b8c68cdd0a9c, State: Initialized, Role: FOLLOWER
I20240501 20:12:03.862882 13800 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 34579 } }
I20240501 20:12:03.863117 13800 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:12:03.863219 13800 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:12:03.863343 13800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:03.865514 13800 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 34579 } }
I20240501 20:12:03.866001 13800 leader_election.cc:302] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdfefe0ff2f6496dba80b8c68cdd0a9c; no voters:
I20240501 20:12:03.866693 13800 leader_election.cc:288] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:12:03.866842 13805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:03.867627 13805 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [term 1 LEADER]: Becoming Leader. State: Replica: cdfefe0ff2f6496dba80b8c68cdd0a9c, State: Running, Role: LEADER
I20240501 20:12:03.868100 13805 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 34579 } }
I20240501 20:12:03.868793 13800 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:12:03.870787 13806 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 34579 } } }
I20240501 20:12:03.870810 13807 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdfefe0ff2f6496dba80b8c68cdd0a9c. Latest consensus state: current_term: 1 leader_uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdfefe0ff2f6496dba80b8c68cdd0a9c" member_type: VOTER last_known_addr { host: "127.28.136.126" port: 34579 } } }
I20240501 20:12:03.871191 13806 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [sys.catalog]: This master's current role is: LEADER
I20240501 20:12:03.871187 13807 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c [sys.catalog]: This master's current role is: LEADER
I20240501 20:12:03.874290 13811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:12:03.880362 13811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:12:03.891233 13811 catalog_manager.cc:1347] Generated new cluster ID: 3dbad32ac303428c8db85238607a11de
I20240501 20:12:03.891356 13811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:12:03.905194 13811 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:12:03.906383 13811 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:12:03.914713 13811 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cdfefe0ff2f6496dba80b8c68cdd0a9c: Generated new TSK 0
I20240501 20:12:03.915503 13811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:12:03.923745 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.65:0
--local_ip_for_outbound_sockets=127.28.136.65
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:12:04.271705 13824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:04.272006 13824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:04.272121 13824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:04.279934 13824 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:04.280069 13824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:04.280223 13824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.65
I20240501 20:12:04.289691 13824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.28.136.65
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.65
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:04.291209 13824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:04.293006 13824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:04.305006 13830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:04.305296 13832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:04.305727 13829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:04.306187 13824 server_base.cc:1034] running on GCE node
I20240501 20:12:04.306710 13824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:04.307595 13824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:04.308820 13824 hybrid_clock.cc:648] HybridClock initialized: now 1714594324308762 us; error 49 us; skew 500 ppm
I20240501 20:12:04.309120 13824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:04.311247 13824 webserver.cc:460] Webserver started at http://127.28.136.65:34329/ using document root <none> and password file <none>
I20240501 20:12:04.311947 13824 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:04.312057 13824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:04.312359 13824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:04.314946 13824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
format_stamp: "Formatted at 2024-05-01 20:12:04 on dist-test-slave-sx0n"
I20240501 20:12:04.315691 13824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
format_stamp: "Formatted at 2024-05-01 20:12:04 on dist-test-slave-sx0n"
I20240501 20:12:04.320667 13824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240501 20:12:04.324019 13838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:04.325438 13824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:04.325600 13824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
format_stamp: "Formatted at 2024-05-01 20:12:04 on dist-test-slave-sx0n"
I20240501 20:12:04.325810 13824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:04.340596 13824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:04.341475 13824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:04.341745 13824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:04.342689 13824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:04.344429 13824 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:04.344530 13824 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:04.344646 13824 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:04.344713 13824 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:04.374682 13824 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.65:42939
I20240501 20:12:04.374748 13950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.65:42939 every 8 connection(s)
I20240501 20:12:04.375947 13824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:12:04.379916 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 13824
I20240501 20:12:04.380136 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:12:04.383826 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.66:0
--local_ip_for_outbound_sockets=127.28.136.66
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:04.389699 13951 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:04.389968 13951 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:04.390759 13951 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:04.435887 13765 ts_manager.cc:194] Registered new tserver with Master: c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939)
I20240501 20:12:04.437783 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.65:47511
W20240501 20:12:04.729720 13955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:04.730168 13955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:04.730322 13955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:04.741851 13955 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:04.741997 13955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:04.742175 13955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.66
I20240501 20:12:04.752758 13955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.28.136.66
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.66
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:04.754388 13955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:04.756551 13955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:04.769662 13961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:04.770184 13962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:04.770589 13964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:04.771049 13955 server_base.cc:1034] running on GCE node
I20240501 20:12:04.771602 13955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:04.772590 13955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:04.773779 13955 hybrid_clock.cc:648] HybridClock initialized: now 1714594324773695 us; error 86 us; skew 500 ppm
I20240501 20:12:04.774087 13955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:04.776276 13955 webserver.cc:460] Webserver started at http://127.28.136.66:38803/ using document root <none> and password file <none>
I20240501 20:12:04.777060 13955 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:04.777180 13955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:04.777493 13955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:04.780056 13955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
format_stamp: "Formatted at 2024-05-01 20:12:04 on dist-test-slave-sx0n"
I20240501 20:12:04.780830 13955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
format_stamp: "Formatted at 2024-05-01 20:12:04 on dist-test-slave-sx0n"
I20240501 20:12:04.786218 13955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 20:12:04.789567 13970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:04.790827 13955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:12:04.790969 13955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
format_stamp: "Formatted at 2024-05-01 20:12:04 on dist-test-slave-sx0n"
I20240501 20:12:04.791154 13955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:04.806968 13955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:04.807796 13955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:04.808044 13955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:04.809041 13955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:04.810539 13955 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:04.810631 13955 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:04.810720 13955 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:04.810773 13955 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:04.848016 13955 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.66:44849
I20240501 20:12:04.848089 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.66:44849 every 8 connection(s)
I20240501 20:12:04.849951 13955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:12:04.851908 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 13955
I20240501 20:12:04.852121 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:12:04.855902 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.67:0
--local_ip_for_outbound_sockets=127.28.136.67
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:04.863585 14083 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:04.863890 14083 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:04.864777 14083 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:04.911885 13765 ts_manager.cc:194] Registered new tserver with Master: c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:04.912873 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.66:56447
W20240501 20:12:05.205425 14086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:05.205720 14086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:05.205866 14086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:05.213506 14086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:05.213642 14086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:05.213827 14086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.67
I20240501 20:12:05.223073 14086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.28.136.67
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.67
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:05.224421 14086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:05.226218 14086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:05.238548 14092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:05.238816 14095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:05.239162 14093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:05.239725 14086 server_base.cc:1034] running on GCE node
I20240501 20:12:05.240346 14086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:05.241492 14086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:05.242710 14086 hybrid_clock.cc:648] HybridClock initialized: now 1714594325242655 us; error 41 us; skew 500 ppm
I20240501 20:12:05.243044 14086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:05.245332 14086 webserver.cc:460] Webserver started at http://127.28.136.67:41129/ using document root <none> and password file <none>
I20240501 20:12:05.246096 14086 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:05.246206 14086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:05.246517 14086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:05.251156 14086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0e898d8ee23441908d1593c561b368ad"
format_stamp: "Formatted at 2024-05-01 20:12:05 on dist-test-slave-sx0n"
I20240501 20:12:05.251941 14086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0e898d8ee23441908d1593c561b368ad"
format_stamp: "Formatted at 2024-05-01 20:12:05 on dist-test-slave-sx0n"
I20240501 20:12:05.257230 14086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:12:05.260670 14101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:05.262038 14086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:12:05.262197 14086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0e898d8ee23441908d1593c561b368ad"
format_stamp: "Formatted at 2024-05-01 20:12:05 on dist-test-slave-sx0n"
I20240501 20:12:05.262398 14086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:05.295033 14086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:05.295941 14086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:05.296223 14086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:05.297266 14086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:05.298744 14086 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:05.298840 14086 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:05.298949 14086 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:05.299016 14086 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:05.331797 14086 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.67:38579
I20240501 20:12:05.331871 14213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.67:38579 every 8 connection(s)
I20240501 20:12:05.333202 14086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:12:05.342600 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 14086
I20240501 20:12:05.342785 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:12:05.345779 14214 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:05.346143 14214 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:05.346472 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.68:0
--local_ip_for_outbound_sockets=127.28.136.68
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:05.347168 14214 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:05.392753 13765 ts_manager.cc:194] Registered new tserver with Master: 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:05.393920 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.67:35257
I20240501 20:12:05.441519 13951 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
W20240501 20:12:05.683290 14218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:05.683566 14218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:05.683702 14218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:05.690992 14218 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:05.691102 14218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:05.691260 14218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.68
I20240501 20:12:05.699895 14218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.28.136.68
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.68
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:05.701176 14218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:05.702816 14218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:05.714161 14226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:05.714813 14218 server_base.cc:1034] running on GCE node
W20240501 20:12:05.716993 14224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:05.719142 14223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:05.719839 14218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:05.720885 14218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:05.722087 14218 hybrid_clock.cc:648] HybridClock initialized: now 1714594325722033 us; error 41 us; skew 500 ppm
I20240501 20:12:05.722381 14218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:05.724431 14218 webserver.cc:460] Webserver started at http://127.28.136.68:45809/ using document root <none> and password file <none>
I20240501 20:12:05.725172 14218 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:05.725291 14218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:05.725586 14218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:05.728081 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "692bb14bda8046b9827f1091b0946088"
format_stamp: "Formatted at 2024-05-01 20:12:05 on dist-test-slave-sx0n"
I20240501 20:12:05.728861 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "692bb14bda8046b9827f1091b0946088"
format_stamp: "Formatted at 2024-05-01 20:12:05 on dist-test-slave-sx0n"
I20240501 20:12:05.733983 14218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 20:12:05.737386 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:05.738647 14218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:12:05.738803 14218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "692bb14bda8046b9827f1091b0946088"
format_stamp: "Formatted at 2024-05-01 20:12:05 on dist-test-slave-sx0n"
I20240501 20:12:05.739013 14218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:05.754621 14218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:05.755482 14218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:05.755759 14218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:05.756723 14218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:05.758272 14218 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:05.758371 14218 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:05.758488 14218 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:05.758561 14218 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:05.788473 14218 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.68:38031
I20240501 20:12:05.788486 14344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.68:38031 every 8 connection(s)
I20240501 20:12:05.789876 14218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:12:05.792132 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 14218
I20240501 20:12:05.792323 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:12:05.795814 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.69:0
--local_ip_for_outbound_sockets=127.28.136.69
--webserver_interface=127.28.136.69
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:05.803493 14345 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:05.803758 14345 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:05.804518 14345 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:05.852452 13765 ts_manager.cc:194] Registered new tserver with Master: 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031)
I20240501 20:12:05.853790 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.68:41577
I20240501 20:12:05.916220 14083 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
W20240501 20:12:06.136245 14348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:06.136547 14348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:06.136691 14348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:06.144387 14348 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:06.144532 14348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:06.144701 14348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.69
I20240501 20:12:06.153806 14348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.28.136.69
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.69
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:06.155133 14348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:06.156868 14348 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:06.168844 14354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:06.169261 14357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:06.169610 14355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:06.169983 14348 server_base.cc:1034] running on GCE node
I20240501 20:12:06.170598 14348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:06.171536 14348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:06.172770 14348 hybrid_clock.cc:648] HybridClock initialized: now 1714594326172681 us; error 42 us; skew 500 ppm
I20240501 20:12:06.173074 14348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:06.175262 14348 webserver.cc:460] Webserver started at http://127.28.136.69:43891/ using document root <none> and password file <none>
I20240501 20:12:06.175997 14348 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:06.176108 14348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:06.176410 14348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:06.179149 14348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
format_stamp: "Formatted at 2024-05-01 20:12:06 on dist-test-slave-sx0n"
I20240501 20:12:06.179914 14348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
format_stamp: "Formatted at 2024-05-01 20:12:06 on dist-test-slave-sx0n"
I20240501 20:12:06.184962 14348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240501 20:12:06.188338 14363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:06.189661 14348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:06.189814 14348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
format_stamp: "Formatted at 2024-05-01 20:12:06 on dist-test-slave-sx0n"
I20240501 20:12:06.190011 14348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:06.210893 14348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:06.211782 14348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:06.212047 14348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:06.213030 14348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:06.214532 14348 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:06.214635 14348 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:06.214749 14348 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:06.214816 14348 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:06.244504 14348 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.69:32991
I20240501 20:12:06.244555 14475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.69:32991 every 8 connection(s)
I20240501 20:12:06.245911 14348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:12:06.252157 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 14348
I20240501 20:12:06.252334 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:12:06.255573 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.70:0
--local_ip_for_outbound_sockets=127.28.136.70
--webserver_interface=127.28.136.70
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:06.259306 14476 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:06.259616 14476 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:06.260429 14476 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:06.302582 13765 ts_manager.cc:194] Registered new tserver with Master: 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:06.304003 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.69:33521
I20240501 20:12:06.397151 14214 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
W20240501 20:12:06.604830 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:06.605124 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:06.605258 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:06.612727 14480 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:06.612891 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:06.613061 14480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.70
I20240501 20:12:06.621970 14480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.28.136.70
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.70
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:06.623260 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:06.624969 14480 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:06.636548 14486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:06.637023 14488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:06.637403 14485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:06.637879 14480 server_base.cc:1034] running on GCE node
I20240501 20:12:06.638440 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:06.639389 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:06.640599 14480 hybrid_clock.cc:648] HybridClock initialized: now 1714594326640539 us; error 53 us; skew 500 ppm
I20240501 20:12:06.640937 14480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:06.643132 14480 webserver.cc:460] Webserver started at http://127.28.136.70:43669/ using document root <none> and password file <none>
I20240501 20:12:06.643932 14480 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:06.644047 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:06.644358 14480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:06.647116 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a5ca96c8a9a74413bff544306af10586"
format_stamp: "Formatted at 2024-05-01 20:12:06 on dist-test-slave-sx0n"
I20240501 20:12:06.647946 14480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a5ca96c8a9a74413bff544306af10586"
format_stamp: "Formatted at 2024-05-01 20:12:06 on dist-test-slave-sx0n"
I20240501 20:12:06.653412 14480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:12:06.657358 14494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:06.658890 14480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:12:06.659150 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a5ca96c8a9a74413bff544306af10586"
format_stamp: "Formatted at 2024-05-01 20:12:06 on dist-test-slave-sx0n"
I20240501 20:12:06.659397 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:06.688088 14480 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:06.689062 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:06.689332 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:06.690271 14480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:06.691743 14480 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:06.691841 14480 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:06.691953 14480 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:06.692028 14480 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:06.721117 14480 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.70:36397
I20240501 20:12:06.721184 14607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.70:36397 every 8 connection(s)
I20240501 20:12:06.722414 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:12:06.731153 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 14480
I20240501 20:12:06.731392 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:12:06.735286 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.71:0
--local_ip_for_outbound_sockets=127.28.136.71
--webserver_interface=127.28.136.71
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:06.735589 14608 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:06.735924 14608 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:06.736801 14608 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:06.797456 13765 ts_manager.cc:194] Registered new tserver with Master: a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:06.798462 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.70:50289
I20240501 20:12:06.857159 14345 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
W20240501 20:12:07.089711 14612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:07.090034 14612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:07.090170 14612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:07.097680 14612 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:07.097819 14612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:07.097981 14612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.71
I20240501 20:12:07.107028 14612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.28.136.71
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.71
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:07.108546 14612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:07.110224 14612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:07.121695 14618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:07.121901 14620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:07.122352 14617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:07.122967 14612 server_base.cc:1034] running on GCE node
I20240501 20:12:07.123556 14612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:07.124498 14612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:07.125698 14612 hybrid_clock.cc:648] HybridClock initialized: now 1714594327125639 us; error 48 us; skew 500 ppm
I20240501 20:12:07.126016 14612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:07.128294 14612 webserver.cc:460] Webserver started at http://127.28.136.71:35265/ using document root <none> and password file <none>
I20240501 20:12:07.129112 14612 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:07.129228 14612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:07.129545 14612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:07.132290 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "3c0f925474374c18a6cd80cff94d4295"
format_stamp: "Formatted at 2024-05-01 20:12:07 on dist-test-slave-sx0n"
I20240501 20:12:07.133109 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "3c0f925474374c18a6cd80cff94d4295"
format_stamp: "Formatted at 2024-05-01 20:12:07 on dist-test-slave-sx0n"
I20240501 20:12:07.138235 14612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240501 20:12:07.141539 14626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:07.142856 14612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:12:07.143005 14612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "3c0f925474374c18a6cd80cff94d4295"
format_stamp: "Formatted at 2024-05-01 20:12:07 on dist-test-slave-sx0n"
I20240501 20:12:07.143194 14612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:07.182325 14612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:07.183218 14612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:07.183482 14612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:07.184468 14612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:07.185987 14612 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:07.186087 14612 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:07.186196 14612 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:07.186264 14612 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:07.215776 14612 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.71:43621
I20240501 20:12:07.215828 14738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.71:43621 every 8 connection(s)
I20240501 20:12:07.217059 14612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:12:07.221201 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 14612
I20240501 20:12:07.221431 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:12:07.225284 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.72:0
--local_ip_for_outbound_sockets=127.28.136.72
--webserver_interface=127.28.136.72
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:07.231088 14739 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:07.231381 14739 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:07.232214 14739 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:07.276399 13765 ts_manager.cc:194] Registered new tserver with Master: 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:07.277449 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.71:56237
I20240501 20:12:07.306739 14476 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
W20240501 20:12:07.567823 14742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:07.568110 14742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:07.568248 14742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:07.576033 14742 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:07.576166 14742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:07.576319 14742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.72
I20240501 20:12:07.585510 14742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.136.72
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.72
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:07.586836 14742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:07.588542 14742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:07.600101 14748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:07.600399 14749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:07.600942 14751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:07.601424 14742 server_base.cc:1034] running on GCE node
I20240501 20:12:07.601984 14742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:07.602885 14742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:07.604092 14742 hybrid_clock.cc:648] HybridClock initialized: now 1714594327604039 us; error 41 us; skew 500 ppm
I20240501 20:12:07.604389 14742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:07.606488 14742 webserver.cc:460] Webserver started at http://127.28.136.72:46005/ using document root <none> and password file <none>
I20240501 20:12:07.607169 14742 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:07.607297 14742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:07.607609 14742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:07.610098 14742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "aeb70769b705493bb5196527311b142b"
format_stamp: "Formatted at 2024-05-01 20:12:07 on dist-test-slave-sx0n"
I20240501 20:12:07.610832 14742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "aeb70769b705493bb5196527311b142b"
format_stamp: "Formatted at 2024-05-01 20:12:07 on dist-test-slave-sx0n"
I20240501 20:12:07.615594 14742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240501 20:12:07.618782 14757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:07.620048 14742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:07.620198 14742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "aeb70769b705493bb5196527311b142b"
format_stamp: "Formatted at 2024-05-01 20:12:07 on dist-test-slave-sx0n"
I20240501 20:12:07.620385 14742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:07.637115 14742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:07.638018 14742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:07.638280 14742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:07.639256 14742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:07.640792 14742 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:07.640889 14742 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:07.641000 14742 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:07.641074 14742 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:07.670540 14742 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.72:39739
I20240501 20:12:07.670599 14869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.72:39739 every 8 connection(s)
I20240501 20:12:07.671851 14742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:12:07.674348 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 14742
I20240501 20:12:07.674597 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:12:07.678345 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.73:0
--local_ip_for_outbound_sockets=127.28.136.73
--webserver_interface=127.28.136.73
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:07.685034 14870 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:07.685339 14870 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:07.686154 14870 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:07.728320 13765 ts_manager.cc:194] Registered new tserver with Master: aeb70769b705493bb5196527311b142b (127.28.136.72:39739)
I20240501 20:12:07.729336 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.72:60523
I20240501 20:12:07.801151 14608 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
W20240501 20:12:08.022055 14873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:08.022379 14873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:08.022502 14873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:08.030153 14873 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:08.030289 14873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:08.030444 14873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.73
I20240501 20:12:08.039580 14873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.73:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.28.136.73
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.73
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:08.040953 14873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:08.042659 14873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:08.054275 14879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:08.054718 14880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:08.055159 14882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:08.055585 14873 server_base.cc:1034] running on GCE node
I20240501 20:12:08.056157 14873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:08.057104 14873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:08.058313 14873 hybrid_clock.cc:648] HybridClock initialized: now 1714594328058262 us; error 40 us; skew 500 ppm
I20240501 20:12:08.058629 14873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:08.063695 14873 webserver.cc:460] Webserver started at http://127.28.136.73:45595/ using document root <none> and password file <none>
I20240501 20:12:08.064436 14873 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:08.064545 14873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:08.064904 14873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:08.067556 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "2a31af84f57c4ef18f0d98ff044229b6"
format_stamp: "Formatted at 2024-05-01 20:12:08 on dist-test-slave-sx0n"
I20240501 20:12:08.068327 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "2a31af84f57c4ef18f0d98ff044229b6"
format_stamp: "Formatted at 2024-05-01 20:12:08 on dist-test-slave-sx0n"
I20240501 20:12:08.073652 14873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240501 20:12:08.077006 14888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:08.078380 14873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:08.078531 14873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "2a31af84f57c4ef18f0d98ff044229b6"
format_stamp: "Formatted at 2024-05-01 20:12:08 on dist-test-slave-sx0n"
I20240501 20:12:08.078722 14873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:08.094772 14873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:08.095614 14873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:08.095875 14873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:08.096908 14873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:08.098443 14873 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:08.098539 14873 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:08.098651 14873 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:08.098717 14873 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:08.128232 14873 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.73:39773
I20240501 20:12:08.128278 15000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.73:39773 every 8 connection(s)
I20240501 20:12:08.129601 14873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240501 20:12:08.133600 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 14873
I20240501 20:12:08.133807 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240501 20:12:08.137437 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.74:0
--local_ip_for_outbound_sockets=127.28.136.74
--webserver_interface=127.28.136.74
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:08.143491 15001 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:08.143793 15001 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:08.144711 15001 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:08.203393 13765 ts_manager.cc:194] Registered new tserver with Master: 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773)
I20240501 20:12:08.204401 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.73:59285
I20240501 20:12:08.280300 14739 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
W20240501 20:12:08.493227 15005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:08.493518 15005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:08.493608 15005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:08.501112 15005 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:08.501228 15005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:08.501351 15005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.74
I20240501 20:12:08.510143 15005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.74:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.28.136.74
--webserver_port=0
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.74
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:08.511384 15005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:08.513085 15005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:08.524627 15010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:08.524977 15011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:08.525411 15013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:08.525947 15005 server_base.cc:1034] running on GCE node
I20240501 20:12:08.526551 15005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:08.527485 15005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:08.528697 15005 hybrid_clock.cc:648] HybridClock initialized: now 1714594328528645 us; error 46 us; skew 500 ppm
I20240501 20:12:08.529018 15005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:08.531303 15005 webserver.cc:460] Webserver started at http://127.28.136.74:42735/ using document root <none> and password file <none>
I20240501 20:12:08.532032 15005 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:08.532145 15005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:08.532435 15005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:12:08.535012 15005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "1899220968384feebca3562bd7741a21"
format_stamp: "Formatted at 2024-05-01 20:12:08 on dist-test-slave-sx0n"
I20240501 20:12:08.535779 15005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "1899220968384feebca3562bd7741a21"
format_stamp: "Formatted at 2024-05-01 20:12:08 on dist-test-slave-sx0n"
I20240501 20:12:08.540818 15005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:12:08.543996 15019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:08.545329 15005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:08.545477 15005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "1899220968384feebca3562bd7741a21"
format_stamp: "Formatted at 2024-05-01 20:12:08 on dist-test-slave-sx0n"
I20240501 20:12:08.545655 15005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:08.560708 15005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:08.561580 15005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:08.561828 15005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:08.562764 15005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:08.564235 15005 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:08.564317 15005 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:08.564397 15005 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:08.564452 15005 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:08.592486 15005 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.74:45465
I20240501 20:12:08.592556 15131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.74:45465 every 8 connection(s)
I20240501 20:12:08.593830 15005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240501 20:12:08.601855 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 15005
I20240501 20:12:08.602036 29217 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240501 20:12:08.606652 15132 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:08.606921 15132 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:08.607724 15132 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:08.651960 13765 ts_manager.cc:194] Registered new tserver with Master: 1899220968384feebca3562bd7741a21 (127.28.136.74:45465)
I20240501 20:12:08.652945 13765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.74:36521
I20240501 20:12:08.657557 29217 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240501 20:12:08.675400 29217 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240501 20:12:08.675559 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 14742
I20240501 20:12:08.683429 14865 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:08.762028 14870 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
I20240501 20:12:08.763391 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 14742
W20240501 20:12:08.777446 13751 connection.cc:535] server connection from 127.28.136.72:60523 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:12:08.777657 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 15005
I20240501 20:12:08.785872 15127 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:08.866568 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 15005
I20240501 20:12:08.880426 29217 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu pid 14873
I20240501 20:12:08.888326 14996 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:12:08.968271 29217 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu with pid 14873
I20240501 20:12:12.734606 13765 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:12:12.741468 13765 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 20:12:12.831529 14280 tablet_service.cc:1467] Processing CreateTablet for tablet c6213df9f24a4e63aec805f7dfacc75d (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.833799 14409 tablet_service.cc:1467] Processing CreateTablet for tablet 60c32a552af1499dab8884bae2287ce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.834846 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.834846 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.838076 14411 tablet_service.cc:1467] Processing CreateTablet for tablet c6213df9f24a4e63aec805f7dfacc75d (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.838598 14411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.838168 14674 tablet_service.cc:1467] Processing CreateTablet for tablet c6213df9f24a4e63aec805f7dfacc75d (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.839560 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.840307 14672 tablet_service.cc:1467] Processing CreateTablet for tablet 60c32a552af1499dab8884bae2287ce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.829697 14410 tablet_service.cc:1467] Processing CreateTablet for tablet 39eb086c2b80482a98ff2baae96b9f60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.841327 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.832705 14279 tablet_service.cc:1467] Processing CreateTablet for tablet 39eb086c2b80482a98ff2baae96b9f60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.843232 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.833289 14278 tablet_service.cc:1467] Processing CreateTablet for tablet 60c32a552af1499dab8884bae2287ce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.848021 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.853241 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.854403 14673 tablet_service.cc:1467] Processing CreateTablet for tablet 39eb086c2b80482a98ff2baae96b9f60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.854964 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.861200 14149 tablet_service.cc:1467] Processing CreateTablet for tablet c6213df9f24a4e63aec805f7dfacc75d (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.862542 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.862524 13885 tablet_service.cc:1467] Processing CreateTablet for tablet 39eb086c2b80482a98ff2baae96b9f60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.864456 13886 tablet_service.cc:1467] Processing CreateTablet for tablet c6213df9f24a4e63aec805f7dfacc75d (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.864992 13884 tablet_service.cc:1467] Processing CreateTablet for tablet 60c32a552af1499dab8884bae2287ce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.865373 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.864723 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 39eb086c2b80482a98ff2baae96b9f60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.866640 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.866905 13886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.860939 14148 tablet_service.cc:1467] Processing CreateTablet for tablet 39eb086c2b80482a98ff2baae96b9f60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.867601 14147 tablet_service.cc:1467] Processing CreateTablet for tablet 60c32a552af1499dab8884bae2287ce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.867993 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.869596 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.864980 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 60c32a552af1499dab8884bae2287ce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.864853 14543 tablet_service.cc:1467] Processing CreateTablet for tablet c6213df9f24a4e63aec805f7dfacc75d (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.870872 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.871156 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.871552 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.888581 15168 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:12.898741 15168 tablet_bootstrap.cc:654] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:12.900117 15168 log.cc:824] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:12.913751 15171 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:12.913213 14018 tablet_service.cc:1467] Processing CreateTablet for tablet c6213df9f24a4e63aec805f7dfacc75d (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.914831 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.917032 14016 tablet_service.cc:1467] Processing CreateTablet for tablet 60c32a552af1499dab8884bae2287ce3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.917500 14016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.919111 15170 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:12.922014 15171 tablet_bootstrap.cc:654] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:12.922107 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 39eb086c2b80482a98ff2baae96b9f60 (DEFAULT_TABLE table=rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:12.922597 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:12.928015 15170 tablet_bootstrap.cc:654] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:12.929487 15171 log.cc:824] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:12.931694 15170 log.cc:824] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:12.940595 15171 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:12.941109 15171 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.000s
I20240501 20:12:12.947547 15170 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:12.948093 15170 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.006s
I20240501 20:12:12.955612 15174 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:12.960036 15174 tablet_bootstrap.cc:654] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:12.961781 15175 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:12.962765 15167 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:12.965767 15170 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:12.966439 15170 raft_consensus.cc:374] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:12.966560 15170 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:12.966787 15171 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:12.967527 15171 raft_consensus.cc:374] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:12.967689 15171 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:12.967525 15170 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:12.968757 15171 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:12.971010 15175 tablet_bootstrap.cc:654] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:12.971433 15168 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:12.971752 15168 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.083s user 0.037s sys 0.000s
I20240501 20:12:12.973752 15167 tablet_bootstrap.cc:654] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:12.975072 15167 log.cc:824] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:12.976305 15174 log.cc:824] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:12.978009 15175 log.cc:824] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:12.978740 15166 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:12.981431 15176 raft_consensus.cc:482] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:12.981703 15176 raft_consensus.cc:504] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:12.982785 15166 tablet_bootstrap.cc:654] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:12.987423 15171 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.046s user 0.018s sys 0.009s
I20240501 20:12:12.988027 15171 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:12.988628 15168 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:12.989115 15170 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.041s user 0.011s sys 0.014s
I20240501 20:12:12.989198 15168 raft_consensus.cc:374] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:12.989306 15168 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:12.989612 15170 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:12.990029 15168 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:12.991605 15170 tablet_bootstrap.cc:654] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:12.992091 15176 leader_election.cc:288] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:12.998414 15171 tablet_bootstrap.cc:654] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.007985 15171 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:13.008301 15171 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.001s
I20240501 20:12:13.009603 15171 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.010104 15171 raft_consensus.cc:374] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.010250 15171 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.010919 15167 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:13.010704 15171 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.011257 15167 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.006s
I20240501 20:12:13.015168 15171 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:12:13.016359 15175 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:13.016677 15175 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.003s
I20240501 20:12:13.018229 15174 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:13.018575 15174 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.003s
I20240501 20:12:13.019034 15171 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:13.022361 15171 tablet_bootstrap.cc:654] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.025177 15166 log.cc:824] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:13.029696 15168 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.058s user 0.014s sys 0.005s
I20240501 20:12:13.030184 15168 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:13.031976 15168 tablet_bootstrap.cc:654] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.036859 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
W20240501 20:12:13.038630 14102 leader_election.cc:341] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.041152 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
W20240501 20:12:13.042359 14105 leader_election.cc:341] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.043154 15170 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:13.043366 15170 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.054s user 0.012s sys 0.025s
I20240501 20:12:13.044649 15170 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.045047 15170 raft_consensus.cc:374] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.045171 15170 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.045464 15170 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.048892 15168 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:13.049079 15168 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 20:12:13.049310 15166 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:13.049721 15166 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.000s
I20240501 20:12:13.050120 15170 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:12:13.050688 15168 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.050982 15168 raft_consensus.cc:374] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.051060 15168 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.051321 15168 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.052037 15168 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:13.052340 15168 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:13.054028 15168 tablet_bootstrap.cc:654] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.050812 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
W20240501 20:12:13.055151 14103 leader_election.cc:341] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.055558 15170 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:13.057498 15170 tablet_bootstrap.cc:654] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.057643 15167 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.058262 15167 raft_consensus.cc:374] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.058351 15167 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.059223 15167 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.061403 15181 raft_consensus.cc:482] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:13.061568 15181 raft_consensus.cc:504] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.062125 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:13.063020 15181 leader_election.cc:288] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:13.063241 15168 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:13.063428 15168 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240501 20:12:13.063704 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:13.064033 13906 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:13.064230 15168 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.064503 15168 raft_consensus.cc:374] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.064577 15168 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.064607 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:13.064939 15168 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.064813 15175 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.065373 15175 raft_consensus.cc:374] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.065490 15175 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.065680 15168 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:12:13.066262 15175 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.066992 15174 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.067521 15174 raft_consensus.cc:374] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.067521 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:13.067634 15174 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.068476 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:13.068884 14563 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
W20240501 20:12:13.069478 14103 leader_election.cc:341] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.069645 14103 leader_election.cc:302] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad; no voters: 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:13.068477 15174 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.070192 15181 raft_consensus.cc:2726] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:13.070537 15167 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.059s user 0.012s sys 0.003s
W20240501 20:12:13.070703 14102 leader_election.cc:341] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:12:13.070956 14103 leader_election.cc:341] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.071043 15167 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:13.072991 15167 tablet_bootstrap.cc:654] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.075956 15170 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:13.076184 15170 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240501 20:12:13.077178 15170 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.077555 15170 raft_consensus.cc:374] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.077641 15170 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.078033 15170 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.080389 15167 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:13.080580 15167 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240501 20:12:13.081449 15170 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:12:13.081679 15167 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.081972 15167 raft_consensus.cc:374] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.082053 15167 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.082314 15167 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.085671 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:13.087284 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:13.087445 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:13.087707 14430 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:13.088047 14694 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:13.087890 15166 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.088428 15166 raft_consensus.cc:374] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.088546 15166 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.088577 14693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:13.089361 15166 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
W20240501 20:12:13.090227 14215 tablet.cc:2369] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:12:13.090277 14104 leader_election.cc:341] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.090543 14104 leader_election.cc:302] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters: 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:13.091003 15181 raft_consensus.cc:2781] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:13.091126 15181 raft_consensus.cc:482] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:13.091235 15181 raft_consensus.cc:3037] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.094250 15181 raft_consensus.cc:504] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.094502 15198 raft_consensus.cc:482] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:13.094803 15198 raft_consensus.cc:504] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.095788 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:13.095968 15174 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.077s user 0.011s sys 0.007s
I20240501 20:12:13.096047 13906 raft_consensus.cc:3037] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:12:13.098009 14104 leader_election.cc:341] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.098837 15198 leader_election.cc:288] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:13.099946 15174 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:13.105127 15175 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.084s user 0.023s sys 0.019s
I20240501 20:12:13.105593 15175 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:13.106024 15167 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.025s user 0.000s sys 0.003s
I20240501 20:12:13.106351 15167 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:13.107334 15175 tablet_bootstrap.cc:654] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.108049 15167 tablet_bootstrap.cc:654] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.110038 15174 tablet_bootstrap.cc:654] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
W20240501 20:12:13.111220 14084 tablet.cc:2369] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:12:13.105181 13906 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:13.114537 15166 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.065s user 0.014s sys 0.006s
I20240501 20:12:13.115012 15166 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:13.116971 15166 tablet_bootstrap.cc:654] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.118470 15174 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:13.118649 15174 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240501 20:12:13.119732 15174 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.120033 15174 raft_consensus.cc:374] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.120131 15174 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.120402 15174 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.120965 15174 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:13.121268 15174 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:13.121838 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
W20240501 20:12:13.122615 14103 leader_election.cc:341] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.122804 15174 tablet_bootstrap.cc:654] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.125221 15175 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:13.125269 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:13.125396 15175 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240501 20:12:13.125480 14430 raft_consensus.cc:3037] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.126214 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:13.126505 14563 raft_consensus.cc:3037] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.126405 15175 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.126662 15175 raft_consensus.cc:374] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.126742 15175 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.126969 15175 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.127544 15175 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:13.128482 14430 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:13.130688 14563 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:13.131814 14103 leader_election.cc:302] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters: 692bb14bda8046b9827f1091b0946088
I20240501 20:12:13.132202 15176 raft_consensus.cc:2781] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election won for term 1
W20240501 20:12:13.133728 13952 tablet.cc:2369] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:12:13.134518 15176 raft_consensus.cc:686] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Becoming Leader. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Running, Role: LEADER
I20240501 20:12:13.135149 15176 consensus_queue.cc:237] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.135892 14693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:13.136157 14693 raft_consensus.cc:3037] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.139546 14693 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:13.141366 15174 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:13.141582 15174 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.001s
I20240501 20:12:13.141772 15181 leader_election.cc:288] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:13.142814 15175 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:13.142387 15174 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.143045 15174 raft_consensus.cc:374] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.143136 15174 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.143409 15174 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.146420 15174 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:12:13.143535 13764 catalog_manager.cc:5484] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e898d8ee23441908d1593c561b368ad (127.28.136.67). New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:13.148514 15167 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:13.148707 15167 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.005s
I20240501 20:12:13.149413 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:13.149526 15167 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.149818 15167 raft_consensus.cc:374] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.149895 15167 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.149956 14169 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:13.150238 15167 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.150960 15167 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:13.153088 15175 tablet_bootstrap.cc:654] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.155217 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:13.155545 13906 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:13.155927 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:13.156188 14430 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:13.157234 13973 leader_election.cc:302] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:13.157792 15200 raft_consensus.cc:2781] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:13.157930 15200 raft_consensus.cc:482] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:13.158038 15200 raft_consensus.cc:3037] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.162276 15200 raft_consensus.cc:504] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.166031 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:13.166280 14169 raft_consensus.cc:3037] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.166337 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:13.166585 14430 raft_consensus.cc:3037] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.169724 14430 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:13.169920 14169 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:13.169993 15196 raft_consensus.cc:482] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:13.165661 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:13.170185 15196 raft_consensus.cc:504] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.170704 13906 raft_consensus.cc:3037] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.170732 15200 leader_election.cc:288] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:13.173669 13906 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:13.174331 13974 leader_election.cc:302] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:13.175428 15200 raft_consensus.cc:2781] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:13.175634 15200 raft_consensus.cc:686] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Becoming Leader. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Running, Role: LEADER
I20240501 20:12:13.176120 15171 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:13.176365 15171 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.157s user 0.003s sys 0.004s
I20240501 20:12:13.176280 15200 consensus_queue.cc:237] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.177779 15171 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.178189 15171 raft_consensus.cc:374] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.178296 15171 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.178618 15171 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.179587 15171 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:12:13.187412 15166 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:13.187628 15166 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.073s user 0.024s sys 0.027s
I20240501 20:12:13.191625 15198 raft_consensus.cc:482] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:13.191838 15198 raft_consensus.cc:504] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.192409 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:13.192646 14694 raft_consensus.cc:3037] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.193199 15198 leader_election.cc:288] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:13.193743 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:13.193980 14038 raft_consensus.cc:3037] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.188727 15166 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.196151 15166 raft_consensus.cc:374] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.196291 15166 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.196928 15166 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.197031 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:13.197561 14692 raft_consensus.cc:2379] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3224a319e2f4cb8a84bf9528209a9fb in current term 1: Already voted for candidate 0e898d8ee23441908d1593c561b368ad in this term.
I20240501 20:12:13.197660 15166 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:12:13.198006 15166 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:13.198071 14693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:13.198889 14694 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:13.199905 15166 tablet_bootstrap.cc:654] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:13.199906 14038 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:13.200963 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:13.201936 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:13.202541 13906 raft_consensus.cc:2379] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3224a319e2f4cb8a84bf9528209a9fb in current term 1: Already voted for candidate 0e898d8ee23441908d1593c561b368ad in this term.
I20240501 20:12:13.204391 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:13.205365 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:13.206169 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:13.206542 14300 raft_consensus.cc:2454] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:13.207466 15166 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:13.207576 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:13.207836 14563 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:13.208348 15166 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 20:12:13.208716 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:13.208922 14563 raft_consensus.cc:3037] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.209234 15166 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.209553 15166 raft_consensus.cc:374] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.209625 15166 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.210438 15196 leader_election.cc:288] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:13.211496 15166 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.211905 14563 raft_consensus.cc:2454] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:13.212771 15166 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:12:13.212788 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:13.213212 14563 raft_consensus.cc:2379] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3224a319e2f4cb8a84bf9528209a9fb in current term 1: Already voted for candidate 0e898d8ee23441908d1593c561b368ad in this term.
W20240501 20:12:13.213984 13972 leader_election.cc:341] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:12:13.214391 13972 leader_election.cc:341] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.220758 13972 leader_election.cc:302] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:13.221388 15198 raft_consensus.cc:2726] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 20:12:13.222514 14740 tablet.cc:2369] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:12:13.228432 14609 tablet.cc:2369] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:12:13.238492 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:13.238803 14169 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:13.241732 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39eb086c2b80482a98ff2baae96b9f60" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:13.242102 14430 raft_consensus.cc:2379] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3224a319e2f4cb8a84bf9528209a9fb in current term 1: Already voted for candidate 0e898d8ee23441908d1593c561b368ad in this term.
I20240501 20:12:13.243132 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:13.243453 14038 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:13.243655 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:13.244714 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:13.244974 13906 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
W20240501 20:12:13.245465 14366 leader_election.cc:341] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.246129 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:13.246381 14300 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:13.246894 14365 leader_election.cc:302] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3c0f925474374c18a6cd80cff94d4295
I20240501 20:12:13.247445 15196 raft_consensus.cc:2781] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:13.247558 15196 raft_consensus.cc:482] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:13.247640 15196 raft_consensus.cc:3037] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.250847 15196 raft_consensus.cc:504] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.252266 15196 leader_election.cc:288] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 election: Requested vote from peers c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
W20240501 20:12:13.252382 14477 tablet.cc:2369] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:12:13.252559 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:13.252810 14169 raft_consensus.cc:3037] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.253314 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:13.253571 14300 raft_consensus.cc:3037] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.255616 14169 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:13.256470 14300 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:13.256585 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:13.257503 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:13.257777 13906 raft_consensus.cc:3037] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.258114 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:13.258399 14038 raft_consensus.cc:3037] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.259243 15175 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:13.260588 13906 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:13.260963 15175 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.133s user 0.033s sys 0.056s
W20240501 20:12:13.261245 14366 leader_election.cc:341] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:13.261255 14038 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:13.261575 14367 leader_election.cc:302] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69; no voters: 3c0f925474374c18a6cd80cff94d4295
I20240501 20:12:13.261979 15196 raft_consensus.cc:2781] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:13.264004 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:13.264034 15175 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.264107 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:13.264317 14563 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:13.264647 15175 raft_consensus.cc:374] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:13.265223 15175 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:13.265187 13764 catalog_manager.cc:5484] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: term changed from 0 to 1, leader changed from <none> to c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66). New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:13.265576 15175 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.266333 15175 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:12:13.269060 15196 raft_consensus.cc:686] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Running, Role: LEADER
I20240501 20:12:13.269726 15196 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.275207 13764 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69). New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:13.293720 15201 raft_consensus.cc:482] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:13.293943 15201 raft_consensus.cc:504] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.296348 15201 leader_election.cc:288] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
W20240501 20:12:13.296772 14346 tablet.cc:2369] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:12:13.310950 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:13.311305 13906 raft_consensus.cc:2379] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 692bb14bda8046b9827f1091b0946088 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:13.317293 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:13.319801 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:13.320137 14562 raft_consensus.cc:2379] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 692bb14bda8046b9827f1091b0946088 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:13.322377 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:13.322932 14694 raft_consensus.cc:2379] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 692bb14bda8046b9827f1091b0946088 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:13.323813 14235 leader_election.cc:302] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 692bb14bda8046b9827f1091b0946088; no voters: 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:13.324359 15201 raft_consensus.cc:3037] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.327149 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:13.327316 15201 raft_consensus.cc:2726] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:13.327502 14169 raft_consensus.cc:2379] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 692bb14bda8046b9827f1091b0946088 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:13.328871 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c32a552af1499dab8884bae2287ce3" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:13.329131 14430 raft_consensus.cc:2379] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 692bb14bda8046b9827f1091b0946088 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:13.369053 15197 raft_consensus.cc:482] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:13.369266 15197 raft_consensus.cc:504] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:13.371739 15197 leader_election.cc:288] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:13.393776 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:13.394102 13906 raft_consensus.cc:2379] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in this term.
I20240501 20:12:13.396705 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:13.397094 14038 raft_consensus.cc:2379] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in this term.
I20240501 20:12:13.402392 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:13.402698 14169 raft_consensus.cc:2379] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in this term.
I20240501 20:12:13.404029 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:13.404349 14300 raft_consensus.cc:2379] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in this term.
I20240501 20:12:13.404970 14628 leader_election.cc:302] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 3c0f925474374c18a6cd80cff94d4295; no voters: 0e898d8ee23441908d1593c561b368ad, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:13.405561 15197 raft_consensus.cc:3037] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.406767 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:13.407003 14562 raft_consensus.cc:2454] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:13.407894 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6213df9f24a4e63aec805f7dfacc75d" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:13.408398 15197 raft_consensus.cc:2726] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:13.564450 15181 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.570966 15200 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.577735 15200 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.591280 15181 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.619458 14300 raft_consensus.cc:3037] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.622258 15181 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.628860 15212 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.630182 15212 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.631023 15181 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.685184 15196 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:13.708154 15229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:13.708479 15229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:13.715875 15229 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:12:13.728096 15200 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.737015 15200 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.740623 15200 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.749521 15196 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.750985 15198 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.762274 15222 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.763379 15222 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:13.768215 14562 raft_consensus.cc:3037] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:13.771555 15222 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 20:12:13.774377 15222 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 542.348 ms
time per row: 0.271174 ms
I20240501 20:12:14.456017 13761 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:12:14.456816 13761 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 20:12:14.466128 13884 tablet_service.cc:1467] Processing CreateTablet for tablet af0f6a4c2e9c43509502a242b596d9da (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.466471 14016 tablet_service.cc:1467] Processing CreateTablet for tablet af0f6a4c2e9c43509502a242b596d9da (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.466650 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.466629 14149 tablet_service.cc:1467] Processing CreateTablet for tablet af0f6a4c2e9c43509502a242b596d9da (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.466907 14016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.467044 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.469734 14280 tablet_service.cc:1467] Processing CreateTablet for tablet af0f6a4c2e9c43509502a242b596d9da (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.470224 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.474627 14543 tablet_service.cc:1467] Processing CreateTablet for tablet af0f6a4c2e9c43509502a242b596d9da (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.475128 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.475577 14672 tablet_service.cc:1467] Processing CreateTablet for tablet af0f6a4c2e9c43509502a242b596d9da (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.476033 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.477298 14411 tablet_service.cc:1467] Processing CreateTablet for tablet af0f6a4c2e9c43509502a242b596d9da (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.477820 14411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.481905 14018 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.482455 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.482951 15340 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:14.485090 15340 tablet_bootstrap.cc:654] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.485316 14410 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.485771 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.485939 14673 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.486348 15341 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:14.486408 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.489287 15341 tablet_bootstrap.cc:654] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.489238 14542 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.489699 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.493867 14147 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.494360 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.494880 14409 tablet_service.cc:1467] Processing CreateTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.495318 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.497236 14674 tablet_service.cc:1467] Processing CreateTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.497298 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.497632 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.497743 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.501703 14016 tablet_service.cc:1467] Processing CreateTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.502154 14016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.502902 15341 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:14.503069 15341 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240501 20:12:14.503962 15341 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.504294 15341 raft_consensus.cc:374] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.504388 15341 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.504700 15341 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.505506 14278 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.505961 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.506379 14148 tablet_service.cc:1467] Processing CreateTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.506891 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.507027 13885 tablet_service.cc:1467] Processing CreateTablet for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.507464 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.509131 14279 tablet_service.cc:1467] Processing CreateTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.509531 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.511757 13886 tablet_service.cc:1467] Processing CreateTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 (DEFAULT_TABLE table=rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:14.512166 13886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:14.514608 15346 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:14.516638 15346 tablet_bootstrap.cc:654] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.525393 15346 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:14.525601 15346 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:12:14.526437 15346 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.526746 15346 raft_consensus.cc:374] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.526837 15346 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.528542 15341 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20240501 20:12:14.528955 15347 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:14.529587 15341 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:14.531155 15341 tablet_bootstrap.cc:654] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.531348 15347 tablet_bootstrap.cc:654] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.532240 15346 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.534612 15344 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:14.537923 15347 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:14.538117 15347 ts_tablet_manager.cc:1367] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240501 20:12:14.538972 15346 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20240501 20:12:14.540498 15346 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:14.545820 15346 tablet_bootstrap.cc:654] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.546013 15344 tablet_bootstrap.cc:654] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.547276 15347 raft_consensus.cc:348] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.547662 15347 raft_consensus.cc:374] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.547762 15347 raft_consensus.cc:729] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.548938 15347 consensus_queue.cc:260] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.550016 15338 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:14.552618 15347 ts_tablet_manager.cc:1398] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240501 20:12:14.553594 15347 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:14.555230 15346 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:14.555294 15347 tablet_bootstrap.cc:654] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.556802 15346 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 20:12:14.557237 15338 tablet_bootstrap.cc:654] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.557690 15339 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:14.558398 15346 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.558728 15346 raft_consensus.cc:374] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.558811 15346 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.559072 15346 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.559366 15344 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:14.559507 15341 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:14.559525 15344 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240501 20:12:14.559571 15339 tablet_bootstrap.cc:654] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.559726 15341 ts_tablet_manager.cc:1367] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.006s
I20240501 20:12:14.560178 15344 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.560395 15344 raft_consensus.cc:374] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.560485 15344 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.560504 15341 raft_consensus.cc:348] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.560827 15341 raft_consensus.cc:374] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.560714 15344 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.562501 15340 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:14.562752 15340 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.080s user 0.013s sys 0.010s
I20240501 20:12:14.562619 15338 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:14.562983 15338 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 20:12:14.563400 15344 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:12:14.563719 15344 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:14.563879 15338 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.564272 15338 raft_consensus.cc:374] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.564381 15338 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.563905 15340 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.564643 15340 raft_consensus.cc:374] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.564805 15340 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.564651 15338 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.565169 15340 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.565969 15340 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:12:14.566309 15340 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:14.566955 15339 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:14.567137 15339 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240501 20:12:14.568037 15340 tablet_bootstrap.cc:654] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.568461 15346 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:12:14.568635 15341 raft_consensus.cc:729] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.569057 15341 consensus_queue.cc:260] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.569550 15339 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.569820 15339 raft_consensus.cc:374] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.569912 15339 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.570179 15339 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.570864 15339 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:12:14.571193 15339 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:14.572299 15346 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:14.572702 15339 tablet_bootstrap.cc:654] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.574625 15344 tablet_bootstrap.cc:654] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.575914 15341 ts_tablet_manager.cc:1398] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 20:12:14.578171 15346 tablet_bootstrap.cc:654] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.580483 15341 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:14.582274 15341 tablet_bootstrap.cc:654] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.585268 15346 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:14.585512 15346 ts_tablet_manager.cc:1367] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 20:12:14.586472 15346 raft_consensus.cc:348] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.588081 15341 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:14.588378 15341 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240501 20:12:14.589769 15340 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:14.590030 15340 ts_tablet_manager.cc:1367] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20240501 20:12:14.590903 15340 raft_consensus.cc:348] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.591284 15340 raft_consensus.cc:374] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.591408 15340 raft_consensus.cc:729] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.591787 15340 consensus_queue.cc:260] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.592643 15340 ts_tablet_manager.cc:1398] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:12:14.597785 15340 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:14.599625 15340 tablet_bootstrap.cc:654] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.601223 15341 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.601651 15341 raft_consensus.cc:374] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.601785 15341 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.602120 15341 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.603981 15341 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 20:12:14.598843 15350 raft_consensus.cc:482] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:14.604344 15346 raft_consensus.cc:374] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.604568 15346 raft_consensus.cc:729] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.604374 15350 raft_consensus.cc:504] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.606096 15346 consensus_queue.cc:260] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.607622 15346 ts_tablet_manager.cc:1398] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 20:12:14.607738 15339 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:14.607925 15339 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.015s
I20240501 20:12:14.608532 15350 leader_election.cc:288] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:14.611099 15345 raft_consensus.cc:482] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:14.611277 15345 raft_consensus.cc:504] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.612600 15345 leader_election.cc:288] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:14.612792 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:14.613086 14037 raft_consensus.cc:2454] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:14.613806 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:14.613801 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:14.614051 15340 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:14.614235 15340 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240501 20:12:14.614316 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
W20240501 20:12:14.614475 14630 leader_election.cc:341] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:14.614589 14300 raft_consensus.cc:2454] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:14.614977 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:14.615231 14563 raft_consensus.cc:2454] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:14.615067 15340 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.615370 15340 raft_consensus.cc:374] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.615466 15340 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
W20240501 20:12:14.615703 14629 leader_election.cc:341] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:14.615829 14629 leader_election.cc:302] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:14.608709 15339 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.616089 15345 raft_consensus.cc:2781] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:14.613622 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:14.616155 15339 raft_consensus.cc:374] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.616195 15345 raft_consensus.cc:482] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:14.616233 15339 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.616300 15345 raft_consensus.cc:3037] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:12:14.617112 14629 leader_election.cc:341] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:14.617091 15339 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.618333 15339 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240501 20:12:14.618649 15339 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:14.619151 15345 raft_consensus.cc:504] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.619777 15347 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:14.619953 15347 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.014s
I20240501 20:12:14.619853 15340 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.621935 15339 tablet_bootstrap.cc:654] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.621763 15347 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.621956 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:14.622035 15347 raft_consensus.cc:374] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.622104 15347 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.622148 14037 raft_consensus.cc:3037] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.622349 15347 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.623430 15340 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 20:12:14.625027 14037 raft_consensus.cc:2454] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.626004 15338 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.063s user 0.005s sys 0.016s
I20240501 20:12:14.626137 15344 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:14.626338 15344 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.012s
I20240501 20:12:14.626374 15338 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:14.627131 15344 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.627465 15344 raft_consensus.cc:374] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.627547 15344 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.627815 15344 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.628078 15338 tablet_bootstrap.cc:654] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.628405 15344 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:12:14.629492 15347 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 20:12:14.630299 15347 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:14.631925 15347 tablet_bootstrap.cc:654] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.633261 15344 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:14.638973 15344 tablet_bootstrap.cc:654] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.639600 15338 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:14.639811 15338 ts_tablet_manager.cc:1367] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240501 20:12:14.640631 15338 raft_consensus.cc:348] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.640985 15338 raft_consensus.cc:374] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.641058 15338 raft_consensus.cc:729] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.641083 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:14.641335 15338 consensus_queue.cc:260] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.641650 13906 raft_consensus.cc:3037] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.642138 15345 leader_election.cc:288] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Requested vote from peers c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:14.642288 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:14.643452 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:14.643743 14563 raft_consensus.cc:3037] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.644522 13906 raft_consensus.cc:2454] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.646684 14563 raft_consensus.cc:2454] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.647190 14628 leader_election.cc:302] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:14.648044 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:14.648290 14300 raft_consensus.cc:3037] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.649196 15338 ts_tablet_manager.cc:1398] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:12:14.649549 15338 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:14.651201 15338 tablet_bootstrap.cc:654] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:14.651271 14300 raft_consensus.cc:2454] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
W20240501 20:12:14.651903 14629 leader_election.cc:341] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:14.652590 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
W20240501 20:12:14.653563 14629 leader_election.cc:341] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:14.656926 15345 raft_consensus.cc:2781] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:14.661211 15338 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:14.661391 15338 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20240501 20:12:14.665320 15338 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.665633 15338 raft_consensus.cc:374] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.665717 15338 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.665993 15338 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.669322 15338 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 20:12:14.681154 15365 raft_consensus.cc:482] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:14.681403 15365 raft_consensus.cc:504] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.682883 15365 leader_election.cc:288] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:14.683127 15381 raft_consensus.cc:482] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:14.683373 15381 raft_consensus.cc:504] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.685196 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:14.685468 13906 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:14.686919 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:14.687220 14037 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:14.687505 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:14.687798 14169 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:14.688405 14629 leader_election.cc:302] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:14.696025 15380 raft_consensus.cc:2781] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:14.696210 15380 raft_consensus.cc:482] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:14.696382 15380 raft_consensus.cc:3037] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.699481 15380 raft_consensus.cc:504] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.705192 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:14.705449 14430 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:14.706586 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:14.707631 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:14.707875 14563 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:14.709793 15339 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:14.709981 15339 ts_tablet_manager.cc:1367] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.091s user 0.020s sys 0.019s
I20240501 20:12:14.710834 15339 raft_consensus.cc:348] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.711140 15339 raft_consensus.cc:374] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.711239 15339 raft_consensus.cc:729] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.711493 15339 consensus_queue.cc:260] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.712330 15339 ts_tablet_manager.cc:1398] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:14.703373 15345 raft_consensus.cc:686] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Becoming Leader. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Running, Role: LEADER
I20240501 20:12:14.713488 15345 consensus_queue.cc:237] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.714255 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:14.714541 14300 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:14.714324 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
W20240501 20:12:14.715248 14496 leader_election.cc:341] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:14.706027 15381 leader_election.cc:288] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
W20240501 20:12:14.708475 14628 leader_election.cc:341] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:14.717319 15347 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:14.717525 15347 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.088s user 0.012s sys 0.016s
I20240501 20:12:14.718345 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:14.718343 15347 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.718580 14169 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:14.718628 15347 raft_consensus.cc:374] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.718726 15347 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.718986 15347 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.719424 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:14.719626 15347 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:12:14.719636 13906 raft_consensus.cc:3037] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.720191 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:14.720393 14169 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:14.722177 13906 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.727840 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:14.727939 15344 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:14.727892 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:14.728094 15344 ts_tablet_manager.cc:1367] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.095s user 0.025s sys 0.020s
I20240501 20:12:14.728101 13906 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:14.728168 13905 raft_consensus.cc:2379] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5ca96c8a9a74413bff544306af10586 in current term 1: Already voted for candidate 3c0f925474374c18a6cd80cff94d4295 in this term.
I20240501 20:12:14.728789 14498 leader_election.cc:302] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters:
I20240501 20:12:14.728847 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:14.729148 14430 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:14.729255 15365 raft_consensus.cc:2781] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:14.729133 15344 raft_consensus.cc:348] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.729363 15365 raft_consensus.cc:482] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:14.729401 15344 raft_consensus.cc:374] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:14.729435 15365 raft_consensus.cc:3037] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.729471 15344 raft_consensus.cc:729] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:14.729681 15344 consensus_queue.cc:260] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.730209 15344 ts_tablet_manager.cc:1398] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:12:14.731356 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:14.731638 14037 raft_consensus.cc:3037] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.732452 15365 raft_consensus.cc:504] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.733392 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:14.733630 14298 raft_consensus.cc:3037] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.734370 14037 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.736344 14298 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.737602 15380 leader_election.cc:288] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:14.738056 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:14.738287 13906 raft_consensus.cc:3037] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.739081 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:14.739162 14035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:14.739326 14038 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:14.739653 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:14.739941 14036 raft_consensus.cc:2379] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5ca96c8a9a74413bff544306af10586 in current term 1: Already voted for candidate 3c0f925474374c18a6cd80cff94d4295 in this term.
I20240501 20:12:14.740401 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:14.740664 14298 raft_consensus.cc:3037] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.740624 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:14.740895 14169 raft_consensus.cc:3037] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.743429 14298 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:14.743640 14169 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:14.743221 13763 catalog_manager.cc:5484] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71). New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:14.744678 15365 leader_election.cc:288] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Requested vote from peers c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:14.746562 13906 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:14.747532 14628 leader_election.cc:302] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:14.747604 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:14.747757 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:14.747848 14430 raft_consensus.cc:3037] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.747944 14169 raft_consensus.cc:3037] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.748121 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:14.748919 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:14.749099 14431 raft_consensus.cc:3037] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.750658 14169 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.750816 14430 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.751703 14431 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:14.751742 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:14.751762 15345 raft_consensus.cc:2781] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:14.751961 14563 raft_consensus.cc:3037] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.752148 15345 raft_consensus.cc:686] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Becoming Leader. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Running, Role: LEADER
I20240501 20:12:14.752539 14497 leader_election.cc:302] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, a5ca96c8a9a74413bff544306af10586; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:14.752549 15345 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.752988 14497 leader_election.cc:302] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters: c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:14.756916 15365 raft_consensus.cc:2781] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:14.759981 14563 raft_consensus.cc:2454] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:14.760141 15350 raft_consensus.cc:2726] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:14.761654 15365 raft_consensus.cc:686] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Becoming Leader. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Running, Role: LEADER
I20240501 20:12:14.762313 15365 consensus_queue.cc:237] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:14.762477 13763 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71). New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:14.768550 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:14.768862 14694 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:14.768666 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6335cd6db4474648bdf39486b7c9bb09" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:14.769191 14692 raft_consensus.cc:3037] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:14.769232 14693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af0f6a4c2e9c43509502a242b596d9da" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:14.771915 14692 raft_consensus.cc:2454] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:14.773474 13763 catalog_manager.cc:5484] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5ca96c8a9a74413bff544306af10586 (127.28.136.70). New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:15.002068 15176 raft_consensus.cc:482] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:15.002310 15176 raft_consensus.cc:504] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:15.003563 15176 leader_election.cc:288] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:15.003937 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:15.003960 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:15.004238 14037 raft_consensus.cc:2379] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e898d8ee23441908d1593c561b368ad in current term 1: Already voted for candidate 3c0f925474374c18a6cd80cff94d4295 in this term.
I20240501 20:12:15.004314 13906 raft_consensus.cc:2379] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e898d8ee23441908d1593c561b368ad in current term 1: Already voted for candidate 3c0f925474374c18a6cd80cff94d4295 in this term.
I20240501 20:12:15.004333 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:15.004547 14431 raft_consensus.cc:2454] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:15.005424 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:15.005442 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:15.005722 14298 raft_consensus.cc:2379] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e898d8ee23441908d1593c561b368ad in current term 1: Already voted for candidate 3c0f925474374c18a6cd80cff94d4295 in this term.
I20240501 20:12:15.005730 14563 raft_consensus.cc:2379] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e898d8ee23441908d1593c561b368ad in current term 1: Already voted for candidate 3c0f925474374c18a6cd80cff94d4295 in this term.
I20240501 20:12:15.005765 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fa6ed23f1f44b5b9fa96d54ed22ee6" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:15.006368 14105 leader_election.cc:302] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7; no voters: 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:15.006664 15176 raft_consensus.cc:3037] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.008652 15176 raft_consensus.cc:2726] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:15.141554 15345 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.158133 15365 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.161502 15365 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:15.162442 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:15.162856 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:12:15.165783 15345 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.171878 15345 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:15.172466 15389 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:12:15.178035 14431 raft_consensus.cc:3037] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.179210 15380 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.181792 15381 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.183521 15380 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.199174 15365 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.203600 15381 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.204072 15381 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.223292 14036 raft_consensus.cc:3037] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.230266 15350 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.232266 15401 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.232417 15365 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.249853 15387 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.252097 15401 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.263907 15380 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:15.267963 15380 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 319.071 ms
time per row: 0.159536 ms
I20240501 20:12:15.802942 13761 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:12:15.803686 13761 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 20:12:15.812865 14410 tablet_service.cc:1467] Processing CreateTablet for tablet 8892d6dce05140babdeeb6c90c366b3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.813179 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 8892d6dce05140babdeeb6c90c366b3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.813241 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 8892d6dce05140babdeeb6c90c366b3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.813360 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.813609 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.813609 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.813984 14018 tablet_service.cc:1467] Processing CreateTablet for tablet 8892d6dce05140babdeeb6c90c366b3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.814424 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.817744 13884 tablet_service.cc:1467] Processing CreateTablet for tablet 8892d6dce05140babdeeb6c90c366b3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.818154 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.821965 14278 tablet_service.cc:1467] Processing CreateTablet for tablet eeaff69149ee4707beba512799b0d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.822374 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.822813 14672 tablet_service.cc:1467] Processing CreateTablet for tablet 8892d6dce05140babdeeb6c90c366b3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.823235 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.823235 14016 tablet_service.cc:1467] Processing CreateTablet for tablet eeaff69149ee4707beba512799b0d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.823624 14016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.825187 14149 tablet_service.cc:1467] Processing CreateTablet for tablet 8892d6dce05140babdeeb6c90c366b3b (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.825632 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.825613 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 59fdeaff3d92460c8ca59eee1144b127 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.825959 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.826359 14542 tablet_service.cc:1467] Processing CreateTablet for tablet eeaff69149ee4707beba512799b0d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.826743 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.829084 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 59fdeaff3d92460c8ca59eee1144b127 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.829085 14279 tablet_service.cc:1467] Processing CreateTablet for tablet 59fdeaff3d92460c8ca59eee1144b127 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.829447 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.829454 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.830391 15488 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:15.833683 15487 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:15.834658 13886 tablet_service.cc:1467] Processing CreateTablet for tablet eeaff69149ee4707beba512799b0d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.835078 13886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.835426 15487 tablet_bootstrap.cc:654] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.837157 13885 tablet_service.cc:1467] Processing CreateTablet for tablet 59fdeaff3d92460c8ca59eee1144b127 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.837538 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.842502 14147 tablet_service.cc:1467] Processing CreateTablet for tablet eeaff69149ee4707beba512799b0d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.842571 14148 tablet_service.cc:1467] Processing CreateTablet for tablet 59fdeaff3d92460c8ca59eee1144b127 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.842978 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.843456 15487 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:15.843607 15487 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:12:15.843573 14674 tablet_service.cc:1467] Processing CreateTablet for tablet eeaff69149ee4707beba512799b0d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.844004 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.844482 15487 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.844770 15487 raft_consensus.cc:374] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.844854 15487 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.845096 14673 tablet_service.cc:1467] Processing CreateTablet for tablet 59fdeaff3d92460c8ca59eee1144b127 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.845101 15487 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.845438 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.847177 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.849747 14411 tablet_service.cc:1467] Processing CreateTablet for tablet eeaff69149ee4707beba512799b0d0df (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.850188 14411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.853120 14409 tablet_service.cc:1467] Processing CreateTablet for tablet 59fdeaff3d92460c8ca59eee1144b127 (DEFAULT_TABLE table=rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:15.853513 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:15.859102 15491 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:15.862751 15491 tablet_bootstrap.cc:654] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.865751 15488 tablet_bootstrap.cc:654] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.868013 15491 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:15.868219 15491 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 20:12:15.869786 15487 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.026s user 0.006s sys 0.004s
I20240501 20:12:15.871297 15491 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.871587 15491 raft_consensus.cc:374] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.871680 15491 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.872721 15489 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:15.874445 15489 tablet_bootstrap.cc:654] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.876075 15491 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.876935 15491 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:12:15.877271 15491 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:15.877949 15488 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:15.878123 15488 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240501 20:12:15.878819 15491 tablet_bootstrap.cc:654] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.878933 15488 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.879211 15488 raft_consensus.cc:374] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.879295 15488 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.879523 15488 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.880120 15488 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:15.880424 15488 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:15.881987 15488 tablet_bootstrap.cc:654] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.889712 15489 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:15.889905 15489 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 20:12:15.890694 15489 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.890992 15489 raft_consensus.cc:374] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.891112 15489 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.891378 15489 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.899345 15488 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:15.899580 15488 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20240501 20:12:15.900449 15488 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.900800 15488 raft_consensus.cc:374] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.900884 15488 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.901201 15488 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.904819 15488 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:12:15.907645 15488 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:15.909399 15488 tablet_bootstrap.cc:654] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.917618 15488 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:15.917789 15488 ts_tablet_manager.cc:1367] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 20:12:15.918578 15488 raft_consensus.cc:348] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.918871 15488 raft_consensus.cc:374] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.918958 15488 raft_consensus.cc:729] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.919252 15488 consensus_queue.cc:260] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.922813 15488 ts_tablet_manager.cc:1398] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:12:15.925859 15487 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:15.927820 15487 tablet_bootstrap.cc:654] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.931413 15487 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:15.931640 15487 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 20:12:15.932473 15487 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.932828 15487 raft_consensus.cc:374] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.932915 15487 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.933202 15487 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.934031 15487 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:15.934432 15487 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:15.936182 15487 tablet_bootstrap.cc:654] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.944423 15487 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:15.944648 15487 ts_tablet_manager.cc:1367] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240501 20:12:15.945552 15487 raft_consensus.cc:348] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.945881 15487 raft_consensus.cc:374] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.945987 15487 raft_consensus.cc:729] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.946332 15487 consensus_queue.cc:260] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.947939 15487 ts_tablet_manager.cc:1398] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240501 20:12:15.957163 14365 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:36397, user_credentials={real_user=slave}} blocked reactor thread for 99246.2us
I20240501 20:12:15.959558 15500 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:15.960964 15504 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:15.961510 15502 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:15.961833 15500 tablet_bootstrap.cc:654] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.965273 15489 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.075s user 0.006s sys 0.066s
I20240501 20:12:15.965602 15489 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:15.967042 15502 tablet_bootstrap.cc:654] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.967167 15489 tablet_bootstrap.cc:654] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.967446 15504 tablet_bootstrap.cc:654] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.971827 15491 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:15.972049 15491 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.095s user 0.060s sys 0.027s
I20240501 20:12:15.972400 15198 raft_consensus.cc:482] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:15.972568 15198 raft_consensus.cc:504] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.972930 15491 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.973253 15491 raft_consensus.cc:374] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.973338 15491 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.973677 15198 leader_election.cc:288] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:15.973598 15491 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.974228 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:15.974323 15491 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:15.974623 15491 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
W20240501 20:12:15.974998 13974 leader_election.cc:341] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:15.975325 14429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:15.975526 14429 raft_consensus.cc:2454] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:15.975919 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:15.976203 15491 tablet_bootstrap.cc:654] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.976430 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:15.976925 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:15.977190 14300 raft_consensus.cc:2454] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
W20240501 20:12:15.977854 13972 leader_election.cc:341] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:15.977824 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:15.978022 14168 raft_consensus.cc:2454] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
W20240501 20:12:15.978076 13973 leader_election.cc:341] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:15.978315 15504 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:15.978468 15504 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240501 20:12:15.978616 13973 leader_election.cc:302] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:15.978916 15500 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:15.979079 15500 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.003s
I20240501 20:12:15.979218 15198 raft_consensus.cc:2781] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:15.979334 15198 raft_consensus.cc:482] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:15.979413 15198 raft_consensus.cc:3037] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.979357 15504 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.979615 15504 raft_consensus.cc:374] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.979707 15504 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.979878 15500 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.979944 15504 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.980185 15500 raft_consensus.cc:374] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.980269 15500 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.980507 15500 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.980916 15502 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:15.981076 15502 ts_tablet_manager.cc:1367] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240501 20:12:15.981608 15500 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:15.981930 15500 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:15.981832 15502 raft_consensus.cc:348] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.982131 15502 raft_consensus.cc:374] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.982239 15502 raft_consensus.cc:729] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.982470 15502 consensus_queue.cc:260] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.982563 15198 raft_consensus.cc:504] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.983050 15502 ts_tablet_manager.cc:1398] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:15.983376 15502 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:15.983994 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
W20240501 20:12:15.984527 13972 leader_election.cc:341] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:15.984848 14429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:15.985076 14429 raft_consensus.cc:3037] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.986347 15504 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:12:15.986464 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:15.986692 13905 raft_consensus.cc:3037] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.986806 15504 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:15.988008 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:15.988231 14692 raft_consensus.cc:3037] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.989310 15504 tablet_bootstrap.cc:654] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.989979 13905 raft_consensus.cc:2454] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:15.990475 15502 tablet_bootstrap.cc:654] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.990813 15198 leader_election.cc:288] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Requested vote from peers 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:15.991093 15500 tablet_bootstrap.cc:654] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:15.991156 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:15.991323 14168 raft_consensus.cc:3037] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.992094 15491 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:15.992274 15491 ts_tablet_manager.cc:1367] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 20:12:15.992763 15489 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:15.992947 15489 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.004s
I20240501 20:12:15.993167 15491 raft_consensus.cc:348] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.993443 15491 raft_consensus.cc:374] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.993592 15491 raft_consensus.cc:729] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.993706 15489 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.993978 15489 raft_consensus.cc:374] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:15.993854 15491 consensus_queue.cc:260] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.994040 15489 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:15.994001 14168 raft_consensus.cc:2454] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:15.994580 15491 ts_tablet_manager.cc:1398] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:15.994482 15489 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.994798 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:15.995010 14300 raft_consensus.cc:3037] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:15.995112 15489 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:12:15.995429 15489 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:15.996803 15480 raft_consensus.cc:482] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:15.996984 15480 raft_consensus.cc:504] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:15.997785 14300 raft_consensus.cc:2454] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:15.998420 15480 leader_election.cc:288] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:15.999577 15502 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:15.999751 15502 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240501 20:12:16.000438 13972 leader_election.cc:302] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters: a5ca96c8a9a74413bff544306af10586
I20240501 20:12:16.000543 15502 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.000855 15502 raft_consensus.cc:374] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:16.000941 15502 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:16.000900 14693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:16.001031 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:16.001238 14431 raft_consensus.cc:2454] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.001214 15502 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.001401 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
W20240501 20:12:16.001534 14497 leader_election.cc:341] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:16.001600 14037 raft_consensus.cc:2454] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.001618 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:16.001900 15198 raft_consensus.cc:2781] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:16.001965 15502 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:16.002278 15502 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:16.002283 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
W20240501 20:12:16.002602 14498 leader_election.cc:341] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:16.003638 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:16.003875 15502 tablet_bootstrap.cc:654] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:16.003893 14168 raft_consensus.cc:2454] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.004480 14497 leader_election.cc:302] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, a5ca96c8a9a74413bff544306af10586, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:16.004909 15480 raft_consensus.cc:2781] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:16.004935 15489 tablet_bootstrap.cc:654] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:16.005028 15480 raft_consensus.cc:482] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:16.005124 15480 raft_consensus.cc:3037] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.006754 15489 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:16.006889 15489 ts_tablet_manager.cc:1367] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240501 20:12:16.007560 15480 raft_consensus.cc:504] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.007624 15489 raft_consensus.cc:348] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.007860 15489 raft_consensus.cc:374] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:16.007939 15489 raft_consensus.cc:729] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:16.008196 15489 consensus_queue.cc:260] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.008816 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:16.008944 15489 ts_tablet_manager.cc:1398] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:16.009013 14298 raft_consensus.cc:3037] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.009887 15500 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:16.010061 15500 ts_tablet_manager.cc:1367] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.001s
I20240501 20:12:16.010839 15500 raft_consensus.cc:348] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.011107 15500 raft_consensus.cc:374] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:16.011168 15500 raft_consensus.cc:729] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:16.011528 14298 raft_consensus.cc:2454] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.011402 15500 consensus_queue.cc:260] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.012089 15500 ts_tablet_manager.cc:1398] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:16.012413 15500 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:16.013257 14692 raft_consensus.cc:2454] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
W20240501 20:12:16.014401 14496 leader_election.cc:341] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:16.014626 15500 tablet_bootstrap.cc:654] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:16.018359 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:16.018604 14431 raft_consensus.cc:3037] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.020035 14429 raft_consensus.cc:2454] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:16.021646 14431 raft_consensus.cc:2454] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.022956 15504 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:16.023166 15504 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.013s
I20240501 20:12:16.023207 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:16.023229 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:16.023434 13905 raft_consensus.cc:3037] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:12:16.023730 14497 leader_election.cc:341] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:16.023845 15500 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:16.024009 15500 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20240501 20:12:16.023988 15504 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.024283 15504 raft_consensus.cc:374] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:16.024354 15504 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:16.024627 15504 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.024818 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:16.025022 14037 raft_consensus.cc:3037] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.024928 15500 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.025187 15500 raft_consensus.cc:374] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:16.025282 15500 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:16.025964 15504 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:16.026285 15504 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:16.026286 13905 raft_consensus.cc:2454] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.025569 15500 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.027539 14037 raft_consensus.cc:2454] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.027669 15500 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:12:16.028036 14498 leader_election.cc:302] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters: 3c0f925474374c18a6cd80cff94d4295
I20240501 20:12:16.028381 15387 raft_consensus.cc:2781] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:16.028429 15504 tablet_bootstrap.cc:654] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:16.028543 15387 raft_consensus.cc:686] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Becoming Leader. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Running, Role: LEADER
I20240501 20:12:16.028942 15387 consensus_queue.cc:237] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.032902 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:16.032776 13761 catalog_manager.cc:5484] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5ca96c8a9a74413bff544306af10586 (127.28.136.70). New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:16.033314 14168 raft_consensus.cc:3037] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.035657 14168 raft_consensus.cc:2454] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.036384 15480 leader_election.cc:288] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Requested vote from peers 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:16.042312 15198 raft_consensus.cc:686] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Becoming Leader. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Running, Role: LEADER
I20240501 20:12:16.042665 15198 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.046126 15504 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:16.046314 15504 ts_tablet_manager.cc:1367] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.008s
I20240501 20:12:16.047128 15504 raft_consensus.cc:348] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.046878 13761 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: term changed from 0 to 1, leader changed from <none> to c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66). New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:16.047417 15504 raft_consensus.cc:374] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:16.047505 15504 raft_consensus.cc:729] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:16.047749 15504 consensus_queue.cc:260] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.047992 15502 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:16.048178 15502 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.011s
I20240501 20:12:16.048404 15504 ts_tablet_manager.cc:1398] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:16.048907 15502 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.049155 15502 raft_consensus.cc:374] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:16.049242 15502 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:16.049489 15502 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.050217 15502 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:16.069669 15480 raft_consensus.cc:482] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:16.069893 15480 raft_consensus.cc:504] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.070982 15480 leader_election.cc:288] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:16.071281 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:16.071282 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:16.071516 13905 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.071540 14431 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.071571 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:16.071826 14692 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.071995 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:16.072230 14168 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.072199 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:16.072324 14497 leader_election.cc:302] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters:
I20240501 20:12:16.072450 14300 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.072791 15480 raft_consensus.cc:2781] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:16.072918 15480 raft_consensus.cc:482] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:16.073006 15480 raft_consensus.cc:3037] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.073019 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:16.073277 14037 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:16.075222 15480 raft_consensus.cc:504] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.076310 15480 leader_election.cc:288] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Requested vote from peers 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:16.076504 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:16.076650 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:16.076777 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:16.076833 14431 raft_consensus.cc:3037] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.076982 14300 raft_consensus.cc:3037] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.076982 13905 raft_consensus.cc:3037] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.079650 14431 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.079665 13905 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.079702 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:16.079936 14037 raft_consensus.cc:3037] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.080310 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:16.080549 14692 raft_consensus.cc:3037] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.080783 14300 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.081408 14497 leader_election.cc:302] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters:
I20240501 20:12:16.081722 15480 raft_consensus.cc:2781] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:16.081745 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaff69149ee4707beba512799b0d0df" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:16.081929 15480 raft_consensus.cc:686] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Becoming Leader. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Running, Role: LEADER
I20240501 20:12:16.081959 14168 raft_consensus.cc:3037] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.082283 15480 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.082456 14037 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.083168 14692 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.084414 14168 raft_consensus.cc:2454] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:16.086450 13763 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5ca96c8a9a74413bff544306af10586 (127.28.136.70). New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:16.087572 15480 raft_consensus.cc:482] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:16.087764 15480 raft_consensus.cc:504] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.089179 15480 leader_election.cc:288] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:16.089331 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:16.089476 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:16.089707 14300 raft_consensus.cc:2379] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5ca96c8a9a74413bff544306af10586 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:16.089721 14431 raft_consensus.cc:2379] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5ca96c8a9a74413bff544306af10586 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:16.090152 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:16.090433 13905 raft_consensus.cc:2379] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5ca96c8a9a74413bff544306af10586 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:16.090843 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:16.090862 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:16.091126 14692 raft_consensus.cc:2379] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5ca96c8a9a74413bff544306af10586 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:16.091146 14168 raft_consensus.cc:2379] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5ca96c8a9a74413bff544306af10586 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:16.091538 14497 leader_election.cc:302] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: a5ca96c8a9a74413bff544306af10586; no voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:16.091853 15480 raft_consensus.cc:3037] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.092281 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8892d6dce05140babdeeb6c90c366b3b" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:16.094152 15480 raft_consensus.cc:2726] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:16.190790 15473 raft_consensus.cc:482] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:16.191089 15473 raft_consensus.cc:504] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:16.192793 15473 leader_election.cc:288] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:16.193145 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:16.193392 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:16.193811 14300 raft_consensus.cc:2379] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate a5ca96c8a9a74413bff544306af10586 in this term.
I20240501 20:12:16.194775 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:16.194841 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:16.195062 14037 raft_consensus.cc:2379] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate a5ca96c8a9a74413bff544306af10586 in this term.
I20240501 20:12:16.195127 14168 raft_consensus.cc:2379] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate a5ca96c8a9a74413bff544306af10586 in this term.
I20240501 20:12:16.195832 14628 leader_election.cc:302] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 3c0f925474374c18a6cd80cff94d4295; no voters: 0e898d8ee23441908d1593c561b368ad, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:16.196298 15345 raft_consensus.cc:3037] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:16.197019 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:16.197149 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fdeaff3d92460c8ca59eee1144b127" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:16.197351 13905 raft_consensus.cc:2379] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate a5ca96c8a9a74413bff544306af10586 in this term.
I20240501 20:12:16.197518 14431 raft_consensus.cc:2379] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c0f925474374c18a6cd80cff94d4295 in current term 1: Already voted for candidate a5ca96c8a9a74413bff544306af10586 in this term.
I20240501 20:12:16.198760 15345 raft_consensus.cc:2726] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:16.434089 15198 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.440881 15480 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.459371 15480 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.462458 15480 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.475849 15387 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.476176 15480 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.486935 15480 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.489441 15365 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.489940 15365 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.499307 15480 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.515192 15198 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.531423 15480 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.537763 15365 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.538350 15365 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.551445 15198 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:16.577483 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:16.577848 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:12:16.583390 15198 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:16.590021 15523 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:12:16.591362 15198 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:16.597065 15553 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 378.49 ms
time per row: 0.189245 ms
I20240501 20:12:17.129271 13760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:12:17.130085 13760 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 20:12:17.139716 13884 tablet_service.cc:1467] Processing CreateTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.139952 14018 tablet_service.cc:1467] Processing CreateTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.140275 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.140422 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.147174 15624 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:17.147430 15623 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:17.147842 14149 tablet_service.cc:1467] Processing CreateTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.148332 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.149385 15623 tablet_bootstrap.cc:654] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.150575 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.151104 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.153208 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.153652 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.158763 14278 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.158984 13886 tablet_service.cc:1467] Processing CreateTablet for tablet 64ded5ed2b224d03a7ac23eb3e749f73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.159250 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.159430 13886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ded5ed2b224d03a7ac23eb3e749f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.160357 14672 tablet_service.cc:1467] Processing CreateTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.160503 14674 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.160810 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.161093 14673 tablet_service.cc:1467] Processing CreateTablet for tablet 64ded5ed2b224d03a7ac23eb3e749f73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.161499 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ded5ed2b224d03a7ac23eb3e749f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.162225 14279 tablet_service.cc:1467] Processing CreateTablet for tablet 64ded5ed2b224d03a7ac23eb3e749f73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.162617 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ded5ed2b224d03a7ac23eb3e749f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.164435 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.164909 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.165509 14148 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.165943 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.169247 14410 tablet_service.cc:1467] Processing CreateTablet for tablet 64ded5ed2b224d03a7ac23eb3e749f73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.169716 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ded5ed2b224d03a7ac23eb3e749f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.170104 14409 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.170526 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.170797 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 64ded5ed2b224d03a7ac23eb3e749f73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.171229 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ded5ed2b224d03a7ac23eb3e749f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.173506 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 64ded5ed2b224d03a7ac23eb3e749f73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.173939 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ded5ed2b224d03a7ac23eb3e749f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.176115 15624 tablet_bootstrap.cc:654] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.169615 14411 tablet_service.cc:1467] Processing CreateTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.177320 14411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.179473 14018 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.179926 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.183048 15627 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:17.183257 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.184875 15627 tablet_bootstrap.cc:654] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.186489 13884 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.187714 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.187927 14149 tablet_service.cc:1467] Processing CreateTablet for tablet 64ded5ed2b224d03a7ac23eb3e749f73 (DEFAULT_TABLE table=rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:17.188160 15627 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:17.188331 15627 ts_tablet_manager.cc:1367] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:12:17.188521 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ded5ed2b224d03a7ac23eb3e749f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:17.189214 15627 raft_consensus.cc:348] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.191401 15628 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:17.191421 15627 raft_consensus.cc:374] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.191745 15627 raft_consensus.cc:729] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.192185 15627 consensus_queue.cc:260] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.200659 15628 tablet_bootstrap.cc:654] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.207908 15632 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:17.210019 15632 tablet_bootstrap.cc:654] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.212494 15630 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:17.214601 15630 tablet_bootstrap.cc:654] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.220002 15632 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:17.221262 15632 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240501 20:12:17.222718 15632 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.225721 15632 raft_consensus.cc:374] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.226212 15632 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.226827 15632 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.229688 15626 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:17.230549 15632 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 20:12:17.231204 15632 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:17.233487 15632 tablet_bootstrap.cc:654] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.236634 15623 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:17.236794 15630 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:17.236980 15630 ts_tablet_manager.cc:1367] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.001s
I20240501 20:12:17.237725 15623 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.011s
I20240501 20:12:17.237862 15630 raft_consensus.cc:348] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.238168 15630 raft_consensus.cc:374] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.238258 15630 raft_consensus.cc:729] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.238495 15630 consensus_queue.cc:260] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.238612 15623 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.238871 15623 raft_consensus.cc:374] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.238965 15623 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.239374 15632 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:17.239229 15623 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.239573 15632 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240501 20:12:17.240263 15630 ts_tablet_manager.cc:1398] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:12:17.240324 15632 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.240623 15632 raft_consensus.cc:374] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.240715 15632 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.240840 15630 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:17.240868 15628 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:17.241025 15628 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.050s user 0.023s sys 0.008s
I20240501 20:12:17.241034 15632 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.241775 15632 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:17.241842 15628 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.242094 15632 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:17.242131 15628 raft_consensus.cc:374] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.242234 15628 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.242473 15628 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.243127 15628 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:17.243378 15626 tablet_bootstrap.cc:654] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.243042 15630 tablet_bootstrap.cc:654] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.245143 15628 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:17.246503 15627 ts_tablet_manager.cc:1398] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.058s user 0.012s sys 0.005s
I20240501 20:12:17.246779 15628 tablet_bootstrap.cc:654] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.246870 15627 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:17.248589 15627 tablet_bootstrap.cc:654] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.249874 15632 tablet_bootstrap.cc:654] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.252614 15634 raft_consensus.cc:482] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:17.252849 15634 raft_consensus.cc:504] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.256850 15624 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:17.257140 15624 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.110s user 0.006s sys 0.016s
I20240501 20:12:17.257334 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
W20240501 20:12:17.258069 14234 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.258203 15624 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.258476 15624 raft_consensus.cc:374] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.258574 15624 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.258868 15624 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.259739 15624 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:17.260051 15624 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:17.260609 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:17.260844 14430 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 0.
I20240501 20:12:17.268209 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
W20240501 20:12:17.268919 14235 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.270253 15624 tablet_bootstrap.cc:654] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.270574 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
W20240501 20:12:17.271226 14236 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.272596 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:17.272657 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
W20240501 20:12:17.273267 14233 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.273411 14233 leader_election.cc:302] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088; no voters: 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb
W20240501 20:12:17.273839 14235 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.274227 15634 leader_election.cc:288] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:17.274847 15634 raft_consensus.cc:2726] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:17.276278 15630 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:17.276501 15630 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.005s
I20240501 20:12:17.277356 15630 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.279101 15630 raft_consensus.cc:374] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.279192 15630 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.280004 15630 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.280856 15626 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:17.282864 15626 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.006s
I20240501 20:12:17.283740 15626 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.284099 15630 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 20:12:17.284802 15626 raft_consensus.cc:374] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.285072 15626 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.285583 15626 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.287549 15630 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:17.289213 15630 tablet_bootstrap.cc:654] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.297602 15624 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:17.297823 15624 ts_tablet_manager.cc:1367] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.004s
I20240501 20:12:17.300676 15627 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:17.300899 15627 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.054s user 0.022s sys 0.004s
I20240501 20:12:17.301550 15627 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.301803 15627 raft_consensus.cc:374] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.301875 15627 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.302114 15627 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.302577 15627 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:17.302918 15627 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:17.305002 15624 raft_consensus.cc:348] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.305186 15628 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:17.305298 15624 raft_consensus.cc:374] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.305377 15624 raft_consensus.cc:729] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.305390 15628 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.019s
I20240501 20:12:17.306102 15628 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.306372 15628 raft_consensus.cc:374] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.306430 15628 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.306659 15628 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.307183 15624 consensus_queue.cc:260] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.307425 15628 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:17.307721 15628 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:17.309234 15628 tablet_bootstrap.cc:654] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.310370 15632 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:17.310528 15632 ts_tablet_manager.cc:1367] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.069s user 0.013s sys 0.018s
I20240501 20:12:17.311237 15632 raft_consensus.cc:348] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.311523 15632 raft_consensus.cc:374] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.311617 15632 raft_consensus.cc:729] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.312901 15632 consensus_queue.cc:260] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.313234 15624 ts_tablet_manager.cc:1398] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240501 20:12:17.313478 15632 ts_tablet_manager.cc:1398] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:12:17.313482 15623 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.076s user 0.002s sys 0.005s
I20240501 20:12:17.313580 15624 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:17.313836 15623 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:17.315156 15624 tablet_bootstrap.cc:654] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.315424 15623 tablet_bootstrap.cc:654] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.317581 15627 tablet_bootstrap.cc:654] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.318882 15626 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240501 20:12:17.319267 15626 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:17.319370 15630 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:17.319535 15630 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240501 20:12:17.320338 15630 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.320621 15630 raft_consensus.cc:374] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.320726 15630 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.320996 15630 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.321403 15626 tablet_bootstrap.cc:654] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.322364 15630 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:17.324553 15626 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:17.325328 15626 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:12:17.327140 15617 raft_consensus.cc:482] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:17.327353 15617 raft_consensus.cc:504] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.328790 15626 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.330116 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:17.330379 14430 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:17.330433 15626 raft_consensus.cc:374] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.330785 15626 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.331184 15617 leader_election.cc:288] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:17.331337 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:17.331583 14692 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:17.331465 15626 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.332311 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
W20240501 20:12:17.333019 13973 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.333696 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:17.334997 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
W20240501 20:12:17.335613 13974 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:12:17.336253 13972 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.349007 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:17.349309 14298 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:17.350564 13972 leader_election.cc:302] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 0e898d8ee23441908d1593c561b368ad, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:17.351182 15617 raft_consensus.cc:2781] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:17.351315 15617 raft_consensus.cc:482] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:17.351385 15617 raft_consensus.cc:3037] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.353732 15626 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240501 20:12:17.354293 15617 raft_consensus.cc:504] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.356089 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:17.356325 14430 raft_consensus.cc:3037] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.358942 15626 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:17.359369 14430 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:17.360697 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:17.360963 14692 raft_consensus.cc:3037] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.362826 15626 tablet_bootstrap.cc:654] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.363795 14692 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:17.365602 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:17.365825 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
W20240501 20:12:17.366510 13974 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:12:17.366789 13972 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.367677 15626 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:17.367688 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
W20240501 20:12:17.368304 13973 leader_election.cc:341] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:17.368544 15617 leader_election.cc:288] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Requested vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:17.371496 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:17.371737 14298 raft_consensus.cc:3037] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.371991 15626 ts_tablet_manager.cc:1367] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 20:12:17.373095 15626 raft_consensus.cc:348] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.374557 15626 raft_consensus.cc:374] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.374545 14298 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:17.374831 15626 raft_consensus.cc:729] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.375064 13972 leader_election.cc:302] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 0e898d8ee23441908d1593c561b368ad, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:17.375734 15617 raft_consensus.cc:2781] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:17.375434 15626 consensus_queue.cc:260] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.377786 15617 raft_consensus.cc:686] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Becoming Leader. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Running, Role: LEADER
I20240501 20:12:17.378206 15617 consensus_queue.cc:237] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.379343 15626 ts_tablet_manager.cc:1398] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:12:17.383387 15623 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:17.383658 15623 ts_tablet_manager.cc:1367] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.009s
I20240501 20:12:17.386147 15627 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:17.385753 13760 catalog_manager.cc:5484] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: term changed from 0 to 1, leader changed from <none> to c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66). New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:17.386314 15627 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.083s user 0.000s sys 0.023s
I20240501 20:12:17.384485 15623 raft_consensus.cc:348] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.387164 15623 raft_consensus.cc:374] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.386977 15627 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.387291 15623 raft_consensus.cc:729] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.387341 15627 raft_consensus.cc:374] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.387472 15627 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.387589 15623 consensus_queue.cc:260] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.387743 15627 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.388420 15623 ts_tablet_manager.cc:1398] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:12:17.388701 15627 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:12:17.388792 15623 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:17.390244 15628 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:17.390424 15628 ts_tablet_manager.cc:1367] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.083s user 0.026s sys 0.029s
I20240501 20:12:17.391628 15624 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:17.391497 15628 raft_consensus.cc:348] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.391759 15628 raft_consensus.cc:374] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.391867 15628 raft_consensus.cc:729] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.391983 15624 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.017s
I20240501 20:12:17.392159 15628 consensus_queue.cc:260] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.393381 15624 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.393654 15628 ts_tablet_manager.cc:1398] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:12:17.393654 15624 raft_consensus.cc:374] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.394230 15624 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.394519 15624 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.395642 15623 tablet_bootstrap.cc:654] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:17.396129 15624 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:12:17.400552 15639 raft_consensus.cc:482] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:17.400754 15639 raft_consensus.cc:504] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.401880 15623 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:17.402118 15623 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240501 20:12:17.403009 15623 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.403355 15623 raft_consensus.cc:374] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:17.403504 15623 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:17.403787 15639 leader_election.cc:288] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:17.403854 15623 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.404620 15623 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:12:17.441982 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:17.445748 15493 raft_consensus.cc:482] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:17.445958 15493 raft_consensus.cc:504] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.447379 15493 leader_election.cc:288] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:17.448940 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:17.449230 14563 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:17.449584 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:17.449815 14037 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:17.450423 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:17.450690 14298 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:17.451635 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:17.451897 14692 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:17.452405 14366 leader_election.cc:302] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:17.452975 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:17.453222 14167 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:17.454095 15493 raft_consensus.cc:2781] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:17.454226 15493 raft_consensus.cc:482] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:17.454306 15493 raft_consensus.cc:3037] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.457726 15493 raft_consensus.cc:504] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.460458 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:17.460712 14298 raft_consensus.cc:3037] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.463455 15631 raft_consensus.cc:482] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:17.463650 15631 raft_consensus.cc:504] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.463874 14298 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:17.465159 15631 leader_election.cc:288] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:17.465516 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:17.465821 14037 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:17.466668 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:17.466897 14563 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:17.467247 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:17.467586 14430 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:17.468106 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:17.468438 14167 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:17.469045 14629 leader_election.cc:302] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:17.469619 15493 leader_election.cc:288] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 election: Requested vote from peers 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:17.469584 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:17.469854 14037 raft_consensus.cc:3037] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.469959 15631 raft_consensus.cc:2781] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:17.470103 15631 raft_consensus.cc:482] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:17.470182 15631 raft_consensus.cc:3037] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.473856 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:17.474123 14692 raft_consensus.cc:3037] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.474082 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:17.474328 14563 raft_consensus.cc:3037] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.478015 14563 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:17.478430 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:17.478925 14298 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:17.481432 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:17.481735 14167 raft_consensus.cc:3037] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.481523 15631 raft_consensus.cc:504] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.484181 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:17.484439 14036 raft_consensus.cc:3037] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.484691 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:17.485008 14563 raft_consensus.cc:3037] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.486047 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:17.486303 13906 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:17.487332 14036 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:17.488437 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:17.488682 14298 raft_consensus.cc:3037] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.488665 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:17.488952 13906 raft_consensus.cc:3037] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.489629 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:17.489892 14169 raft_consensus.cc:3037] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.490598 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:17.490855 13905 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:17.491015 15631 leader_election.cc:288] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:17.491770 13906 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:17.492610 14563 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:17.492717 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae91bfd01694bea81cda5ffc9e7bc77" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:17.491832 14298 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:17.492934 13906 raft_consensus.cc:3037] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.493162 14628 leader_election.cc:302] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:17.493314 14169 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:17.493506 15631 raft_consensus.cc:2781] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:17.493774 15631 raft_consensus.cc:686] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Becoming Leader. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Running, Role: LEADER
I20240501 20:12:17.494122 15631 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:17.495455 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e21fb0e6bac46fbbd29bbd2f1bf43b4" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:17.495626 13906 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:17.495715 14430 raft_consensus.cc:3037] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.496510 14367 leader_election.cc:302] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters:
I20240501 20:12:17.496645 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:17.497221 14563 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 0.
I20240501 20:12:17.498657 14430 raft_consensus.cc:2454] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:17.498821 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:17.499086 14431 raft_consensus.cc:2379] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c12d6cddb7f242b6af650a10bbc8eb69 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:17.500411 15493 raft_consensus.cc:2781] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:17.500528 15493 raft_consensus.cc:686] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Running, Role: LEADER
I20240501 20:12:17.500527 14167 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:17.500859 15493 consensus_queue.cc:237] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.501111 14037 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:17.501535 14692 raft_consensus.cc:2454] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:17.501870 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:17.502506 14298 raft_consensus.cc:2379] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c12d6cddb7f242b6af650a10bbc8eb69 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:17.503247 15176 raft_consensus.cc:482] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:17.503425 15176 raft_consensus.cc:504] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:17.503913 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:17.504182 14692 raft_consensus.cc:2379] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c12d6cddb7f242b6af650a10bbc8eb69 in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:17.504804 15176 leader_election.cc:288] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:17.505584 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:17.506556 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:17.506788 14431 raft_consensus.cc:2379] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e898d8ee23441908d1593c561b368ad in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:17.506880 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:17.506942 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:17.507167 14692 raft_consensus.cc:2379] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e898d8ee23441908d1593c561b368ad in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:17.507387 14563 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:17.507541 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:17.507849 14298 raft_consensus.cc:2379] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0e898d8ee23441908d1593c561b368ad in current term 1: Already voted for candidate c3224a319e2f4cb8a84bf9528209a9fb in this term.
I20240501 20:12:17.508581 14103 leader_election.cc:302] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:17.508693 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:17.508961 13906 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:17.509058 15176 raft_consensus.cc:3037] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.509321 13841 leader_election.cc:302] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:17.509794 15639 raft_consensus.cc:3037] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.511224 15176 raft_consensus.cc:2726] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:17.512640 15639 raft_consensus.cc:2726] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:17.515182 13760 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71). New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:17.515926 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ded5ed2b224d03a7ac23eb3e749f73" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:17.516171 14169 raft_consensus.cc:2454] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 1.
I20240501 20:12:17.517874 13761 catalog_manager.cc:5484] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69). New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:17.786410 15617 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.818581 14563 raft_consensus.cc:3037] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:17.821614 15617 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.854818 15617 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.859380 15617 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.864256 15198 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.866369 15198 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.946569 15631 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.946892 15493 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.948467 15345 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:17.948465 15670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:17.948911 15670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:12:17.955929 15493 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:17.959204 15670 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:12:17.969662 15345 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.973640 15631 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:17.989293 15493 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:18.007396 15699 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:18.008553 15700 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:18.050235 15708 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:18.053086 15493 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:18.055135 15709 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
W20240501 20:12:18.378998 14495 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:44849, user_credentials={real_user=slave}} blocked reactor thread for 52133.5us
W20240501 20:12:18.380173 14103 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:36397, user_credentials={real_user=slave}} blocked reactor thread for 137328us
W20240501 20:12:18.591190 13974 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:42939, user_credentials={real_user=slave}} blocked reactor thread for 49786.7us
INSERT report
rows total: 2000
time total: 508.702 ms
time per row: 0.254351 ms
I20240501 20:12:18.715566 13760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:12:18.716301 13760 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 20:12:18.725821 13884 tablet_service.cc:1467] Processing CreateTablet for tablet 7d09003d74074293b86f2390f25ee58a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.726356 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d09003d74074293b86f2390f25ee58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.727102 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 7d09003d74074293b86f2390f25ee58a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.727631 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d09003d74074293b86f2390f25ee58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.727905 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 56c262e138874431a07f6d0bfcb516bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.728394 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.729622 14279 tablet_service.cc:1467] Processing CreateTablet for tablet 7d09003d74074293b86f2390f25ee58a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.730091 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d09003d74074293b86f2390f25ee58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.730046 14411 tablet_service.cc:1467] Processing CreateTablet for tablet 7d09003d74074293b86f2390f25ee58a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.730468 14411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d09003d74074293b86f2390f25ee58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.733246 14149 tablet_service.cc:1467] Processing CreateTablet for tablet 7d09003d74074293b86f2390f25ee58a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.733476 14673 tablet_service.cc:1467] Processing CreateTablet for tablet 7d09003d74074293b86f2390f25ee58a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.733744 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d09003d74074293b86f2390f25ee58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.733908 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d09003d74074293b86f2390f25ee58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.734958 14018 tablet_service.cc:1467] Processing CreateTablet for tablet 7d09003d74074293b86f2390f25ee58a (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.735378 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d09003d74074293b86f2390f25ee58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.737136 14148 tablet_service.cc:1467] Processing CreateTablet for tablet 56c262e138874431a07f6d0bfcb516bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.737571 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.737955 13886 tablet_service.cc:1467] Processing CreateTablet for tablet 56c262e138874431a07f6d0bfcb516bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.738467 13886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.741861 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 56c262e138874431a07f6d0bfcb516bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.742326 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.742998 14147 tablet_service.cc:1467] Processing CreateTablet for tablet 48770e5bebdf4d0b8766738665f7b560 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.743484 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.745303 14409 tablet_service.cc:1467] Processing CreateTablet for tablet 56c262e138874431a07f6d0bfcb516bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.745743 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.749270 14672 tablet_service.cc:1467] Processing CreateTablet for tablet 56c262e138874431a07f6d0bfcb516bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.749575 14410 tablet_service.cc:1467] Processing CreateTablet for tablet 48770e5bebdf4d0b8766738665f7b560 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.749825 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.749974 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.753186 14674 tablet_service.cc:1467] Processing CreateTablet for tablet 48770e5bebdf4d0b8766738665f7b560 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.753639 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.759435 13885 tablet_service.cc:1467] Processing CreateTablet for tablet 48770e5bebdf4d0b8766738665f7b560 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.759923 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.764678 14018 tablet_service.cc:1467] Processing CreateTablet for tablet 48770e5bebdf4d0b8766738665f7b560 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.765255 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.766273 14278 tablet_service.cc:1467] Processing CreateTablet for tablet 48770e5bebdf4d0b8766738665f7b560 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.766743 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.772441 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 56c262e138874431a07f6d0bfcb516bf (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.772951 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.777166 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 48770e5bebdf4d0b8766738665f7b560 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:18.777626 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:18.778235 15782 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:18.780488 15782 tablet_bootstrap.cc:654] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.781690 15780 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:18.783854 15780 tablet_bootstrap.cc:654] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.784551 15778 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:18.786260 15778 tablet_bootstrap.cc:654] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.786262 15784 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:18.787859 15783 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:18.790009 15784 tablet_bootstrap.cc:654] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.791855 15783 tablet_bootstrap.cc:654] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.794880 15784 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:18.795826 15784 ts_tablet_manager.cc:1367] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240501 20:12:18.797149 15783 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:18.798436 15784 raft_consensus.cc:348] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.799005 15783 ts_tablet_manager.cc:1367] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240501 20:12:18.800030 15783 raft_consensus.cc:348] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.800995 15784 raft_consensus.cc:374] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.801259 15784 raft_consensus.cc:729] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.801805 15784 consensus_queue.cc:260] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.802258 15783 raft_consensus.cc:374] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.802539 15783 raft_consensus.cc:729] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.803213 15783 consensus_queue.cc:260] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.805414 15784 ts_tablet_manager.cc:1398] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240501 20:12:18.813470 15784 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:18.817750 15779 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:18.819239 15784 tablet_bootstrap.cc:654] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.824131 15779 tablet_bootstrap.cc:654] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.830447 15781 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:18.846371 15781 tablet_bootstrap.cc:654] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.849081 15780 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:18.849385 15780 ts_tablet_manager.cc:1367] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.068s user 0.014s sys 0.015s
I20240501 20:12:18.850347 15780 raft_consensus.cc:348] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.850746 15780 raft_consensus.cc:374] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.850881 15780 raft_consensus.cc:729] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.851006 15783 ts_tablet_manager.cc:1398] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.052s user 0.000s sys 0.005s
I20240501 20:12:18.851352 15783 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:18.851583 15778 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:18.851603 15780 consensus_queue.cc:260] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.851815 15778 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.067s user 0.021s sys 0.007s
I20240501 20:12:18.852320 15780 ts_tablet_manager.cc:1398] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:12:18.852636 15780 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:18.853085 15778 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.853521 15778 raft_consensus.cc:374] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.853685 15778 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.854004 15778 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.854729 15780 tablet_bootstrap.cc:654] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.856570 15782 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:18.857375 15778 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:12:18.857633 15783 tablet_bootstrap.cc:654] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.858529 15782 ts_tablet_manager.cc:1367] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.080s user 0.007s sys 0.022s
I20240501 20:12:18.858820 15778 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:18.859405 15782 raft_consensus.cc:348] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.859758 15782 raft_consensus.cc:374] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.859896 15782 raft_consensus.cc:729] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.860335 15779 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:18.860220 15782 consensus_queue.cc:260] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.860507 15779 ts_tablet_manager.cc:1367] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240501 20:12:18.861095 15778 tablet_bootstrap.cc:654] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.861127 15782 ts_tablet_manager.cc:1398] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:12:18.861433 15782 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:18.862143 15781 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:18.862881 15784 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:18.863049 15782 tablet_bootstrap.cc:654] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.863165 15781 ts_tablet_manager.cc:1367] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240501 20:12:18.863148 15784 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.001s
I20240501 20:12:18.864293 15784 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.864599 15784 raft_consensus.cc:374] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.864688 15784 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.865032 15784 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.865723 15780 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:18.865916 15780 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240501 20:12:18.866092 15781 raft_consensus.cc:348] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.866685 15780 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.866955 15780 raft_consensus.cc:374] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.867030 15780 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.867445 15781 raft_consensus.cc:374] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.867511 15784 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:12:18.867352 15780 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.867657 15781 raft_consensus.cc:729] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.868063 15780 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:12:18.868383 15780 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:18.866982 15779 raft_consensus.cc:348] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.868599 15779 raft_consensus.cc:374] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.868710 15779 raft_consensus.cc:729] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.869108 15781 consensus_queue.cc:260] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.869865 15784 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:18.870733 15781 ts_tablet_manager.cc:1398] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:12:18.870863 15780 tablet_bootstrap.cc:654] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.870057 15779 consensus_queue.cc:260] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.872584 15784 tablet_bootstrap.cc:654] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.872756 15779 ts_tablet_manager.cc:1398] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240501 20:12:18.874132 15779 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:18.874688 15782 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:18.874941 15782 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20240501 20:12:18.875766 15782 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.876123 15782 raft_consensus.cc:374] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.876259 15782 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.876605 15782 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.877307 15781 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:18.878655 15783 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:18.878880 15783 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240501 20:12:18.879484 15782 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:12:18.880426 15782 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:18.881124 15784 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:18.881294 15784 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20240501 20:12:18.881989 15784 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.882270 15784 raft_consensus.cc:374] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.882297 15779 tablet_bootstrap.cc:654] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.882351 15784 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.882418 15782 tablet_bootstrap.cc:654] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.882617 15784 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.883848 15784 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:18.884933 15780 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:18.885102 15780 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 20:12:18.885397 15782 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:18.885586 15782 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:12:18.886595 15780 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.886857 15780 raft_consensus.cc:374] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.888038 15780 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.888370 15780 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.889611 15780 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:12:18.891059 15783 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.891366 15783 raft_consensus.cc:374] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.891458 15783 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.891714 15783 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.891774 15782 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.892053 15782 raft_consensus.cc:374] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.892146 15782 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.892391 15781 tablet_bootstrap.cc:654] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.892390 15783 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240501 20:12:18.892436 15782 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.893407 15783 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:18.896036 15782 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:12:18.897863 15781 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:18.898787 15778 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:18.898996 15778 ts_tablet_manager.cc:1367] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.003s
I20240501 20:12:18.899224 15783 tablet_bootstrap.cc:654] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.899825 15778 raft_consensus.cc:348] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.900193 15778 raft_consensus.cc:374] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.900292 15778 raft_consensus.cc:729] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.900614 15778 consensus_queue.cc:260] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.901537 15783 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:18.901675 15783 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:12:18.902236 15779 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:18.902369 15783 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.902648 15783 raft_consensus.cc:374] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.902745 15783 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.902688 15779 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.005s
I20240501 20:12:18.903084 15781 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 20:12:18.904085 15778 ts_tablet_manager.cc:1398] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:12:18.905009 15781 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.904028 15779 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.905635 15779 raft_consensus.cc:374] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.905776 15779 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.905958 15781 raft_consensus.cc:374] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.906226 15781 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.906529 15779 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.907061 15778 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:18.907184 15781 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.908056 15779 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:12:18.911216 15779 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:18.911455 15781 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:12:18.911813 15781 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:18.912930 15783 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.914312 15783 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240501 20:12:18.914575 15779 tablet_bootstrap.cc:654] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.916409 15781 tablet_bootstrap.cc:654] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.917037 15778 tablet_bootstrap.cc:654] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:18.919853 15778 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:18.920051 15778 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:12:18.921728 15778 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.922099 15778 raft_consensus.cc:374] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.922205 15778 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.922574 15778 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.925136 15779 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:18.925304 15779 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20240501 20:12:18.925984 15779 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.926242 15779 raft_consensus.cc:374] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.926343 15779 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.926225 15778 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:12:18.926652 15779 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.927392 15779 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:18.928706 15781 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:18.928954 15781 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.012s
I20240501 20:12:18.929934 15781 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.930243 15781 raft_consensus.cc:374] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:18.930346 15781 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:18.930645 15781 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.931409 15781 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:12:18.981696 15752 raft_consensus.cc:482] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:18.981946 15752 raft_consensus.cc:504] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.983326 15752 leader_election.cc:288] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:18.983649 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:18.983940 13906 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:18.983960 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:18.984221 14298 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:18.984344 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:18.984555 14167 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:18.984656 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:18.984943 14037 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:18.984934 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:18.985178 14431 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:18.985455 14495 leader_election.cc:302] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:18.985761 15717 raft_consensus.cc:2781] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:18.985857 15717 raft_consensus.cc:482] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:18.985783 14693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:18.985958 15717 raft_consensus.cc:3037] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:18.986022 14693 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 0.
I20240501 20:12:18.988173 15717 raft_consensus.cc:504] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.989126 15717 leader_election.cc:288] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:18.989269 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:18.989420 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:18.989500 13906 raft_consensus.cc:3037] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:18.989645 14037 raft_consensus.cc:3037] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:18.989630 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:18.989758 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:18.989852 14298 raft_consensus.cc:3037] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:18.989941 14431 raft_consensus.cc:3037] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:18.991084 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:18.991317 14692 raft_consensus.cc:3037] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:18.992455 14298 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:18.992455 14037 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:18.992564 13906 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:18.993099 14498 leader_election.cc:302] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:18.993304 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770e5bebdf4d0b8766738665f7b560" candidate_uuid: "a5ca96c8a9a74413bff544306af10586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:18.993463 15717 raft_consensus.cc:2781] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:18.993528 14167 raft_consensus.cc:3037] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:18.993607 15717 raft_consensus.cc:686] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Becoming Leader. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Running, Role: LEADER
I20240501 20:12:18.994040 14692 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:18.993908 15717 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.995316 14431 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:18.995712 15696 raft_consensus.cc:482] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:18.995919 15696 raft_consensus.cc:504] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:18.996124 14167 raft_consensus.cc:2454] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5ca96c8a9a74413bff544306af10586 in term 1.
I20240501 20:12:18.997866 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:18.998086 13906 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:18.999122 15696 leader_election.cc:288] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:18.999559 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:18.999822 14563 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:18.999861 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:19.000067 14037 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:18.998955 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:19.000460 14627 leader_election.cc:302] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:18.999518 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:19.000787 15696 raft_consensus.cc:2781] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:19.000800 14298 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:19.000892 15696 raft_consensus.cc:482] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:19.000977 15696 raft_consensus.cc:3037] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.000459 14431 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:19.001933 13760 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5ca96c8a9a74413bff544306af10586 (127.28.136.70). New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:18.998708 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:19.003504 14167 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:19.003638 15696 raft_consensus.cc:504] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:19.004884 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:19.004907 14037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:19.005095 13906 raft_consensus.cc:3037] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.005113 14037 raft_consensus.cc:3037] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.005285 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:19.005496 14167 raft_consensus.cc:3037] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.007860 13906 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:19.008042 14037 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:19.010154 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:19.010385 14431 raft_consensus.cc:3037] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.010720 15696 leader_election.cc:288] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:19.010725 14167 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:19.011063 14629 leader_election.cc:302] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:19.011217 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:19.011349 15699 raft_consensus.cc:2781] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:19.011462 14563 raft_consensus.cc:3037] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.012562 14431 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:19.011255 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c262e138874431a07f6d0bfcb516bf" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:19.012817 14298 raft_consensus.cc:3037] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.014215 14563 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:19.014317 15754 raft_consensus.cc:482] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:19.014485 15754 raft_consensus.cc:504] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:19.015447 14298 raft_consensus.cc:2454] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:19.015767 15754 leader_election.cc:288] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:19.016222 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:19.016404 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:19.016492 13906 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:19.016621 14692 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:19.016633 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:19.016860 14300 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:19.017488 15699 raft_consensus.cc:686] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Becoming Leader. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Running, Role: LEADER
I20240501 20:12:19.017535 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:19.017753 14563 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:19.017812 14103 leader_election.cc:302] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69; no voters:
I20240501 20:12:19.017872 15699 consensus_queue.cc:237] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:19.019289 15754 raft_consensus.cc:2781] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:19.019412 15754 raft_consensus.cc:482] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:19.019522 15754 raft_consensus.cc:3037] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.019474 14035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:19.019726 14035 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:19.020001 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:19.020232 14431 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:19.022465 15754 raft_consensus.cc:504] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:19.024596 14035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:19.024832 14035 raft_consensus.cc:3037] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.024685 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:19.024976 14563 raft_consensus.cc:3037] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.026912 14035 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:19.027992 14563 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:19.028015 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:19.028242 14298 raft_consensus.cc:3037] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.028726 15754 leader_election.cc:288] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:19.029527 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:19.029536 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:19.029783 14692 raft_consensus.cc:3037] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.029825 14431 raft_consensus.cc:3037] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.030784 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d09003d74074293b86f2390f25ee58a" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:19.031029 13906 raft_consensus.cc:3037] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:19.032574 14692 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:19.032881 14431 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:19.033012 14104 leader_election.cc:302] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:19.033358 15754 raft_consensus.cc:2781] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:19.033522 15754 raft_consensus.cc:686] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Becoming Leader. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Running, Role: LEADER
I20240501 20:12:19.033805 13906 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:19.033882 15754 consensus_queue.cc:237] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:19.034976 14298 raft_consensus.cc:2454] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:19.032685 13760 catalog_manager.cc:5484] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71). New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:19.037899 13759 catalog_manager.cc:5484] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e898d8ee23441908d1593c561b368ad (127.28.136.67). New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:19.401279 15752 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.412628 15809 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.414917 15810 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.429564 15809 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.438655 15810 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.451380 15699 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:19.459092 15813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:19.459442 15813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:19.469662 15813 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:12:19.472977 15699 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.476713 15699 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.477236 15699 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.482583 15717 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.493793 15810 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.508436 15810 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.516129 15717 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.548215 15717 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.550563 15717 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.571348 15752 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:12:19.595337 15810 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:19.595773 15810 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
W20240501 20:12:19.934087 14103 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:36397, user_credentials={real_user=slave}} blocked reactor thread for 163861us
INSERT report
rows total: 2000
time total: 440.494 ms
time per row: 0.220247 ms
I20240501 20:12:20.210109 13764 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:12:20.210886 13764 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 20:12:20.220249 13885 tablet_service.cc:1467] Processing CreateTablet for tablet 58e126e98f1349cdb3e532ff49d3e926 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.220978 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e126e98f1349cdb3e532ff49d3e926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.221549 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 58e126e98f1349cdb3e532ff49d3e926 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.221966 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e126e98f1349cdb3e532ff49d3e926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.225481 14149 tablet_service.cc:1467] Processing CreateTablet for tablet 58e126e98f1349cdb3e532ff49d3e926 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.225909 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e126e98f1349cdb3e532ff49d3e926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.225860 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 58e126e98f1349cdb3e532ff49d3e926 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.226289 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e126e98f1349cdb3e532ff49d3e926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.229247 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 58e126e98f1349cdb3e532ff49d3e926 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.229704 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e126e98f1349cdb3e532ff49d3e926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.234042 14411 tablet_service.cc:1467] Processing CreateTablet for tablet 58e126e98f1349cdb3e532ff49d3e926 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.234045 14673 tablet_service.cc:1467] Processing CreateTablet for tablet 58e126e98f1349cdb3e532ff49d3e926 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.234488 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e126e98f1349cdb3e532ff49d3e926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.234488 14411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e126e98f1349cdb3e532ff49d3e926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.238221 15924 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:20.239678 13884 tablet_service.cc:1467] Processing CreateTablet for tablet 77ef10335c1642818694b42ab006a66f (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.240144 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.241448 15924 tablet_bootstrap.cc:654] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.243160 15925 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:20.245278 15925 tablet_bootstrap.cc:654] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.247150 15923 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:20.249228 15923 tablet_bootstrap.cc:654] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.251386 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 77ef10335c1642818694b42ab006a66f (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.251895 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.255869 14147 tablet_service.cc:1467] Processing CreateTablet for tablet 77ef10335c1642818694b42ab006a66f (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.256343 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.269150 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 77ef10335c1642818694b42ab006a66f (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.269522 15930 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:20.269724 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.271384 15930 tablet_bootstrap.cc:654] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.278725 15930 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:20.280452 15930 ts_tablet_manager.cc:1367] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 20:12:20.281749 15926 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:20.283545 15925 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:20.283895 15925 ts_tablet_manager.cc:1367] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.013s
I20240501 20:12:20.284896 15925 raft_consensus.cc:348] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.285116 14542 tablet_service.cc:1467] Processing CreateTablet for tablet 77ef10335c1642818694b42ab006a66f (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.285355 15925 raft_consensus.cc:374] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.285454 15925 raft_consensus.cc:729] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.285549 15926 tablet_bootstrap.cc:654] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.285586 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.285761 15925 consensus_queue.cc:260] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.282496 15930 raft_consensus.cc:348] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.286403 15930 raft_consensus.cc:374] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.286510 15930 raft_consensus.cc:729] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.286794 15930 consensus_queue.cc:260] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.287516 15930 ts_tablet_manager.cc:1398] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 20:12:20.291847 15925 ts_tablet_manager.cc:1398] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20240501 20:12:20.293390 15923 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:20.293596 15923 ts_tablet_manager.cc:1367] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.011s
I20240501 20:12:20.295336 14674 tablet_service.cc:1467] Processing CreateTablet for tablet 77ef10335c1642818694b42ab006a66f (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.295797 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.296517 15923 raft_consensus.cc:348] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.296864 15923 raft_consensus.cc:374] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.296947 15923 raft_consensus.cc:729] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.297175 15923 consensus_queue.cc:260] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.297837 15923 ts_tablet_manager.cc:1398] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:12:20.298141 15923 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:20.299780 15923 tablet_bootstrap.cc:654] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.304780 15926 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:20.305011 15926 ts_tablet_manager.cc:1367] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20240501 20:12:20.305852 15926 raft_consensus.cc:348] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.306208 15926 raft_consensus.cc:374] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.306341 15926 raft_consensus.cc:729] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.306668 15926 consensus_queue.cc:260] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.306871 14411 tablet_service.cc:1467] Processing CreateTablet for tablet 77ef10335c1642818694b42ab006a66f (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.307340 14411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.310138 15926 ts_tablet_manager.cc:1398] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:12:20.314271 15923 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:20.314464 15923 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20240501 20:12:20.315269 15923 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.315544 15923 raft_consensus.cc:374] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.315622 15923 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.315892 15923 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.317500 15923 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 20:12:20.321185 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.321718 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.321861 14278 tablet_service.cc:1467] Processing CreateTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.322288 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.326594 14409 tablet_service.cc:1467] Processing CreateTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.327040 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.329176 15925 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:20.330113 15930 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:20.330933 15925 tablet_bootstrap.cc:654] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.333495 15930 tablet_bootstrap.cc:654] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.335589 15926 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:20.338758 15765 raft_consensus.cc:482] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:20.338959 15765 raft_consensus.cc:504] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.340409 15765 leader_election.cc:288] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:20.342787 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:20.343015 13904 raft_consensus.cc:2454] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:20.344082 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:20.344884 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:20.345139 14431 raft_consensus.cc:2454] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
W20240501 20:12:20.346278 13973 leader_election.cc:341] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.347028 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:20.347323 14167 raft_consensus.cc:2454] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:20.349647 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:20.350315 14674 tablet_service.cc:1467] Processing CreateTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.350821 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.351263 13973 leader_election.cc:302] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3c0f925474374c18a6cd80cff94d4295
W20240501 20:12:20.352401 13972 leader_election.cc:341] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.353228 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.353251 13884 tablet_service.cc:1467] Processing CreateTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.353297 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:20.353675 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.353698 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:12:20.355116 13972 leader_election.cc:341] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.356304 15765 raft_consensus.cc:2781] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:20.356328 14149 tablet_service.cc:1467] Processing CreateTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 (DEFAULT_TABLE table=rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:20.356438 15765 raft_consensus.cc:482] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:20.356528 15765 raft_consensus.cc:3037] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.356823 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:20.364264 15923 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:20.366024 15923 tablet_bootstrap.cc:654] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.368234 15765 raft_consensus.cc:504] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.369150 15932 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:20.370549 15936 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:20.372176 15924 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:20.372208 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:20.372359 15924 ts_tablet_manager.cc:1367] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.134s user 0.035s sys 0.063s
I20240501 20:12:20.372443 13904 raft_consensus.cc:3037] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.372864 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:20.373080 14167 raft_consensus.cc:3037] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.373248 15924 raft_consensus.cc:348] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.373534 15924 raft_consensus.cc:374] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.373636 15924 raft_consensus.cc:729] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.375484 13904 raft_consensus.cc:2454] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:20.376000 15924 consensus_queue.cc:260] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.376561 14167 raft_consensus.cc:2454] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:20.378429 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:20.378656 14298 raft_consensus.cc:3037] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.379020 15936 tablet_bootstrap.cc:654] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.380435 15924 ts_tablet_manager.cc:1398] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:12:20.380816 15924 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:20.381091 15932 tablet_bootstrap.cc:654] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.382035 14298 raft_consensus.cc:2454] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:20.382483 15924 tablet_bootstrap.cc:654] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.383824 13972 leader_election.cc:302] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:20.384184 15769 raft_consensus.cc:2781] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:20.384379 15923 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:20.384547 15923 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20240501 20:12:20.385387 15923 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.385670 15923 raft_consensus.cc:374] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.385775 15923 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.386082 15923 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.388753 15924 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:20.388944 15924 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:12:20.389787 15924 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.390108 15924 raft_consensus.cc:374] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.390172 15924 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.392437 15924 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.395417 15924 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:12:20.395787 15924 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:20.398540 15924 tablet_bootstrap.cc:654] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.404563 15924 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:20.404774 15924 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:12:20.405668 15924 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.407836 15924 raft_consensus.cc:374] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.408145 15924 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.408702 15924 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.410838 15924 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:12:20.412453 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
W20240501 20:12:20.413409 13972 leader_election.cc:341] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.414367 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:20.414634 14431 raft_consensus.cc:3037] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.418733 14431 raft_consensus.cc:2454] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:20.434257 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e126e98f1349cdb3e532ff49d3e926" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
W20240501 20:12:20.435079 13973 leader_election.cc:341] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.436017 15765 leader_election.cc:288] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:20.449252 15765 raft_consensus.cc:482] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:20.449607 15765 raft_consensus.cc:504] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.451488 15940 raft_consensus.cc:482] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:20.451653 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:20.451681 15940 raft_consensus.cc:504] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
W20240501 20:12:20.452296 13973 leader_election.cc:341] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.453102 15940 leader_election.cc:288] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:20.454156 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:20.454195 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
W20240501 20:12:20.454756 14234 leader_election.cc:341] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.454844 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
W20240501 20:12:20.454949 14235 leader_election.cc:341] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:12:20.455394 14235 leader_election.cc:341] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.455966 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
W20240501 20:12:20.456552 14236 leader_election.cc:341] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.456713 14236 leader_election.cc:302] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 692bb14bda8046b9827f1091b0946088; no voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:20.457100 15940 raft_consensus.cc:2726] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:20.457278 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
W20240501 20:12:20.457901 14235 leader_election.cc:341] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.461279 15930 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:20.461593 15930 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.141s user 0.004s sys 0.000s
I20240501 20:12:20.462393 15926 tablet_bootstrap.cc:654] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.462704 15930 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.463255 15930 raft_consensus.cc:374] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.463380 15930 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.463856 15930 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.466344 15765 leader_election.cc:288] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:20.467595 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
W20240501 20:12:20.468688 13973 leader_election.cc:341] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.469203 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
W20240501 20:12:20.469959 13974 leader_election.cc:341] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.470098 15925 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:20.470806 15926 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:20.471072 15926 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.136s user 0.003s sys 0.001s
I20240501 20:12:20.471125 15925 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.142s user 0.005s sys 0.025s
I20240501 20:12:20.471966 15926 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.472349 15926 raft_consensus.cc:374] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.472471 15926 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.473153 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:20.473286 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:20.473439 14298 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:20.473595 15925 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.473975 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:20.474013 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:20.474231 14036 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 0.
I20240501 20:12:20.474112 15926 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.475294 15926 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:12:20.475620 15926 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:20.475762 15925 raft_consensus.cc:374] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.475926 15925 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
W20240501 20:12:20.476092 13973 leader_election.cc:341] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.476217 13973 leader_election.cc:302] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 692bb14bda8046b9827f1091b0946088, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69
I20240501 20:12:20.476307 15925 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.476634 15769 raft_consensus.cc:686] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Becoming Leader. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Running, Role: LEADER
I20240501 20:12:20.477149 15925 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:12:20.477285 15926 tablet_bootstrap.cc:654] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.477116 15769 consensus_queue.cc:237] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.477875 15923 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.093s user 0.020s sys 0.048s
I20240501 20:12:20.478271 15932 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:20.478495 15932 ts_tablet_manager.cc:1367] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.018s
I20240501 20:12:20.479421 15932 raft_consensus.cc:348] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.480167 15925 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:20.480316 15932 raft_consensus.cc:374] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.482362 15925 tablet_bootstrap.cc:654] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.482563 15932 raft_consensus.cc:729] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.482975 15932 consensus_queue.cc:260] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.483923 15932 ts_tablet_manager.cc:1398] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:12:20.485220 15936 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:20.485478 15936 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.115s user 0.003s sys 0.024s
I20240501 20:12:20.486256 15926 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:20.486481 15926 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 20:12:20.486451 15936 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.486894 15936 raft_consensus.cc:374] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.487041 15936 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.487362 15926 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.487394 15936 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.487731 15926 raft_consensus.cc:374] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.487876 15926 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.488212 15926 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.489013 15926 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:12:20.489053 15936 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:20.489531 15932 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:20.489924 15936 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:20.491811 15936 tablet_bootstrap.cc:654] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.494757 15932 tablet_bootstrap.cc:654] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.497267 15943 raft_consensus.cc:2726] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:20.497467 15925 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:20.497665 15925 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 20:12:20.498562 15925 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.498880 15925 raft_consensus.cc:374] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.499110 15925 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.499922 15930 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.038s user 0.003s sys 0.000s
I20240501 20:12:20.500070 15932 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:20.500321 15930 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:20.500361 15932 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
W20240501 20:12:20.501360 13972 leader_election.cc:341] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:20.500649 15925 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.502415 15925 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:12:20.502564 15932 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.503271 15932 raft_consensus.cc:374] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.503374 15932 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.503650 15932 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.505249 15930 tablet_bootstrap.cc:654] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.506350 15932 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 20:12:20.506776 15932 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:20.507977 15936 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:20.507793 13765 catalog_manager.cc:5484] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: term changed from 0 to 1, leader changed from <none> to c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66). New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:20.508189 15936 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 20:12:20.509001 15936 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.509387 15936 raft_consensus.cc:374] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.509517 15936 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.509821 15936 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.510483 15936 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:20.510843 15936 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:20.512313 15936 tablet_bootstrap.cc:654] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.514750 15932 tablet_bootstrap.cc:654] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:20.516750 15932 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:20.516908 15932 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240501 20:12:20.517473 15932 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.517674 15932 raft_consensus.cc:374] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.517763 15932 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.517997 15932 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.518535 15932 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:20.520944 15936 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:20.521121 15936 ts_tablet_manager.cc:1367] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 20:12:20.522101 15936 raft_consensus.cc:348] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.522548 15936 raft_consensus.cc:374] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.522670 15936 raft_consensus.cc:729] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.522990 15936 consensus_queue.cc:260] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.524363 15936 ts_tablet_manager.cc:1398] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:20.525735 15930 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:20.525946 15930 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20240501 20:12:20.526626 15930 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.526834 15930 raft_consensus.cc:374] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:20.526932 15930 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:20.527161 15930 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.527732 15930 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:12:20.646229 15879 raft_consensus.cc:482] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:20.646414 15879 raft_consensus.cc:504] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.647711 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:20.647729 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:20.647794 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:20.647861 15879 leader_election.cc:288] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:20.647965 14036 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:20.647982 13904 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:20.648012 14167 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:20.648671 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:20.648825 14629 leader_election.cc:302] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:20.648809 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:20.648957 14563 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:20.649012 14298 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:20.649130 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:20.649288 15879 raft_consensus.cc:2781] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:20.649392 15879 raft_consensus.cc:482] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:20.649374 14431 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 0.
I20240501 20:12:20.649478 15879 raft_consensus.cc:3037] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.651371 15879 raft_consensus.cc:504] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.652395 15879 leader_election.cc:288] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:20.652499 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:20.652578 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:20.652770 13904 raft_consensus.cc:3037] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.652832 14036 raft_consensus.cc:3037] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.653110 14431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:20.653182 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:20.653302 14431 raft_consensus.cc:3037] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.653393 14167 raft_consensus.cc:3037] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.655440 13904 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:20.655468 14036 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:20.655867 14431 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:20.655906 14167 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:20.656319 14629 leader_election.cc:302] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:20.656423 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:20.656661 14563 raft_consensus.cc:3037] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.656647 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ef10335c1642818694b42ab006a66f" candidate_uuid: "3c0f925474374c18a6cd80cff94d4295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:20.656886 14298 raft_consensus.cc:3037] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.657883 15879 raft_consensus.cc:2781] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:20.658269 15879 raft_consensus.cc:686] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Becoming Leader. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Running, Role: LEADER
I20240501 20:12:20.658830 15879 consensus_queue.cc:237] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:20.659425 14563 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:20.659438 14298 raft_consensus.cc:2454] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0f925474374c18a6cd80cff94d4295 in term 1.
I20240501 20:12:20.662561 13764 catalog_manager.cc:5484] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71). New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:20.702351 15708 raft_consensus.cc:482] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:20.702596 15708 raft_consensus.cc:504] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.703662 15708 leader_election.cc:288] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:20.703953 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:20.703939 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:20.703976 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:20.704182 14563 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:20.704221 14692 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:20.704223 14298 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:20.704767 14365 leader_election.cc:302] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586; no voters:
I20240501 20:12:20.704891 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:20.704964 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:20.705144 14167 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:20.705175 15708 raft_consensus.cc:2781] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:20.705097 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:20.705205 14036 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:20.705264 15708 raft_consensus.cc:482] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:20.705353 15708 raft_consensus.cc:3037] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.705333 13904 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:20.708161 15708 raft_consensus.cc:504] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.709300 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:20.709362 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:20.709514 14563 raft_consensus.cc:3037] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.709578 14692 raft_consensus.cc:3037] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.709856 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:20.710081 14036 raft_consensus.cc:3037] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.710070 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:20.710254 14167 raft_consensus.cc:3037] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.712211 14563 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:20.712235 14692 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:20.712594 14036 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:20.712646 15708 leader_election.cc:288] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 election: Requested vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:20.712926 13904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:20.713078 14167 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:20.713160 13904 raft_consensus.cc:3037] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.713459 14366 leader_election.cc:302] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3b9f58142e4a4ccd9d0cbafc97d459c7, a5ca96c8a9a74413bff544306af10586, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:20.713503 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bedd9f3d3c4ee2890206cb43591675" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:20.713759 14298 raft_consensus.cc:3037] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.713892 15708 raft_consensus.cc:2781] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:20.714052 15708 raft_consensus.cc:686] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Running, Role: LEADER
I20240501 20:12:20.714598 15708 consensus_queue.cc:237] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:20.715268 13904 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:20.716298 14298 raft_consensus.cc:2454] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 1.
I20240501 20:12:20.725732 13764 catalog_manager.cc:5484] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69). New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:20.877538 14692 raft_consensus.cc:3037] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.881295 15769 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:20.882941 14563 raft_consensus.cc:3037] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:20.885392 15769 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.021307 15769 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.035939 15971 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.038275 15769 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.052351 15959 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.055115 15879 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.061542 15769 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.110148 15961 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.113438 15959 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.120146 15879 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.130810 15977 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.134095 15959 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.159586 15961 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:21.163313 15963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:21.163637 15963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:21.171821 15963 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 20:12:21.222996 15961 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.229643 15708 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.233345 15708 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:21.233772 16013 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 576.869 ms
time per row: 0.288435 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:12:22.823367 13765 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 20:12:22.824112 13765 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240501 20:12:22.834738 14149 tablet_service.cc:1467] Processing CreateTablet for tablet fe1592bebd6b4a858987847446f15a16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.835162 14411 tablet_service.cc:1467] Processing CreateTablet for tablet fe1592bebd6b4a858987847446f15a16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.835299 14149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.835577 14411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.838127 14018 tablet_service.cc:1467] Processing CreateTablet for tablet 3808b45349de4959b929a99735e5db53 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.838567 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.838783 13885 tablet_service.cc:1467] Processing CreateTablet for tablet 3808b45349de4959b929a99735e5db53 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.839210 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.839780 14673 tablet_service.cc:1467] Processing CreateTablet for tablet 3808b45349de4959b929a99735e5db53 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.840193 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.841750 13884 tablet_service.cc:1467] Processing CreateTablet for tablet fe1592bebd6b4a858987847446f15a16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.842154 13884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.845157 14674 tablet_service.cc:1467] Processing CreateTablet for tablet fe1592bebd6b4a858987847446f15a16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.845156 14542 tablet_service.cc:1467] Processing CreateTablet for tablet fe1592bebd6b4a858987847446f15a16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.845572 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.845590 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.848268 14672 tablet_service.cc:1467] Processing CreateTablet for tablet 1840c628fc114b309da99d3c8a91656a (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.848687 14672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.849246 14017 tablet_service.cc:1467] Processing CreateTablet for tablet fe1592bebd6b4a858987847446f15a16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.849727 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.853356 14280 tablet_service.cc:1467] Processing CreateTablet for tablet fe1592bebd6b4a858987847446f15a16 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.853837 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.858103 14409 tablet_service.cc:1467] Processing CreateTablet for tablet 3808b45349de4959b929a99735e5db53 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.858573 14409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.861166 14410 tablet_service.cc:1467] Processing CreateTablet for tablet 1840c628fc114b309da99d3c8a91656a (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.861598 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.861867 14016 tablet_service.cc:1467] Processing CreateTablet for tablet 1840c628fc114b309da99d3c8a91656a (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.862391 14016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.866735 14147 tablet_service.cc:1467] Processing CreateTablet for tablet 3808b45349de4959b929a99735e5db53 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.867209 14147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.869731 14278 tablet_service.cc:1467] Processing CreateTablet for tablet 3808b45349de4959b929a99735e5db53 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.870201 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.873203 14279 tablet_service.cc:1467] Processing CreateTablet for tablet 1840c628fc114b309da99d3c8a91656a (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.873615 14279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.873821 16086 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:22.875497 16082 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:22.878595 14543 tablet_service.cc:1467] Processing CreateTablet for tablet 3808b45349de4959b929a99735e5db53 (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.878677 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 1840c628fc114b309da99d3c8a91656a (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.879077 14543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.879571 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.887421 16082 tablet_bootstrap.cc:654] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.889262 13885 tablet_service.cc:1467] Processing CreateTablet for tablet 1840c628fc114b309da99d3c8a91656a (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.889710 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.897218 16086 tablet_bootstrap.cc:654] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.897476 16085 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:22.899204 16085 tablet_bootstrap.cc:654] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.905510 16082 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:22.905778 16082 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20240501 20:12:22.905929 16085 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:22.906174 16085 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 20:12:22.906646 16082 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.907020 16082 raft_consensus.cc:374] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.907164 16082 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.907120 14148 tablet_service.cc:1467] Processing CreateTablet for tablet 1840c628fc114b309da99d3c8a91656a (DEFAULT_TABLE table=rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:12:22.907476 16082 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.907768 14148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:22.908079 16082 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:12:22.909389 16082 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:22.910912 16082 tablet_bootstrap.cc:654] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.913148 16082 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:22.913383 16082 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:12:22.914134 16082 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.914587 16082 raft_consensus.cc:374] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.914724 16082 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.915153 16086 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:22.915036 16082 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.915616 16086 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240501 20:12:22.915796 16082 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:22.917903 16086 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.919000 16084 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:22.921147 16084 tablet_bootstrap.cc:654] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.925222 16086 raft_consensus.cc:374] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.925431 16086 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.925797 16086 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.926661 16085 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.926882 16085 raft_consensus.cc:374] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.926963 16085 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.927184 16085 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.927733 16085 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.021s user 0.000s sys 0.000s
I20240501 20:12:22.928050 16085 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:22.929023 16086 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 20:12:22.929344 16086 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:22.931119 16086 tablet_bootstrap.cc:654] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.932885 16085 tablet_bootstrap.cc:654] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.938139 16086 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:22.938345 16086 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:12:22.939312 16085 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:22.939365 16084 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:22.939566 16084 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240501 20:12:22.939872 16085 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 20:12:22.940524 16086 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.940536 16084 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:22.940873 16084 raft_consensus.cc:374] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.940995 16084 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.941267 16084 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:22.941764 16085 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.942114 16086 raft_consensus.cc:374] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.942476 16085 raft_consensus.cc:374] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.942262 16086 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.942842 16085 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.943109 16086 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.943727 16085 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.945008 16084 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:12:22.945784 16086 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:12:22.950433 16084 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:22.952682 16085 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20240501 20:12:22.954107 16084 tablet_bootstrap.cc:654] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.955885 16084 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:22.956056 16084 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:12:22.956663 16084 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.956934 16084 raft_consensus.cc:374] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.957015 16084 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.957253 16084 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.957962 16084 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:22.964316 16101 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:22.965741 16101 tablet_bootstrap.cc:654] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.967495 16101 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:22.967528 16085 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295: Bootstrap starting.
I20240501 20:12:22.967739 16101 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:12:22.968617 16101 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.969044 16101 raft_consensus.cc:374] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.969166 16085 tablet_bootstrap.cc:654] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.969220 16101 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.969630 16101 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.977885 16085 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295: No bootstrap required, opened a new log
I20240501 20:12:22.977916 16101 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20240501 20:12:22.978152 16085 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 20:12:22.978827 16086 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69: Bootstrap starting.
I20240501 20:12:22.980543 16086 tablet_bootstrap.cc:654] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.982987 16085 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:22.983991 16101 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:22.985652 16101 tablet_bootstrap.cc:654] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.986600 16085 raft_consensus.cc:374] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.986907 16085 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0f925474374c18a6cd80cff94d4295, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.988301 16084 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:22.989398 16085 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:22.990188 16084 tablet_bootstrap.cc:654] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:22.993000 16084 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: No bootstrap required, opened a new log
I20240501 20:12:22.993207 16084 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:12:22.993690 16085 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
I20240501 20:12:22.994011 16084 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.994309 16084 raft_consensus.cc:374] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:22.994401 16084 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: FOLLOWER
I20240501 20:12:22.994618 16084 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:22.995263 16084 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:23.003980 16086 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69: No bootstrap required, opened a new log
I20240501 20:12:23.004251 16086 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.010s
I20240501 20:12:23.005599 16086 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.005999 16086 raft_consensus.cc:374] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.006139 16086 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.006489 16086 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.007462 16082 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:23.009163 16082 tablet_bootstrap.cc:654] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:23.009946 16086 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:12:23.015206 16101 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:23.015451 16101 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.005s
I20240501 20:12:23.017725 16101 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.018049 16101 raft_consensus.cc:374] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.018163 16101 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.018386 16096 raft_consensus.cc:482] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:23.018451 16082 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: No bootstrap required, opened a new log
I20240501 20:12:23.018596 16082 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 20:12:23.018458 16101 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.018585 16096 raft_consensus.cc:504] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.019071 16101 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:12:23.019436 16101 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:23.019928 16096 leader_election.cc:288] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:23.020256 16082 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.020526 16082 raft_consensus.cc:374] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.020606 16082 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.020864 16082 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.021270 16101 tablet_bootstrap.cc:654] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:23.021629 14561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:23.022260 16082 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240501 20:12:23.023101 13840 leader_election.cc:341] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:23.023329 16101 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088: No bootstrap required, opened a new log
I20240501 20:12:23.023475 16101 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:12:23.024194 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:23.024317 16101 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.024554 16101 raft_consensus.cc:374] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.024652 16101 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.024914 16101 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.025207 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:23.025458 14692 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 0.
I20240501 20:12:23.025540 16101 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240501 20:12:23.027240 13841 leader_election.cc:341] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:23.027755 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:23.027989 14297 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 0.
I20240501 20:12:23.029059 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:23.029290 14036 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 0.
I20240501 20:12:23.030274 13839 leader_election.cc:302] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, a5ca96c8a9a74413bff544306af10586
I20240501 20:12:23.030921 16096 raft_consensus.cc:2781] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:23.031054 16096 raft_consensus.cc:482] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:23.031167 16096 raft_consensus.cc:3037] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.034435 16096 raft_consensus.cc:504] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.036018 14561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:23.036080 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
W20240501 20:12:23.036592 13840 leader_election.cc:341] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:12:23.036862 13841 leader_election.cc:341] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:23.037820 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:23.038044 14297 raft_consensus.cc:3037] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.042402 16105 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
W20240501 20:12:23.042421 14104 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:43621, user_credentials={real_user=slave}} blocked reactor thread for 160672us
I20240501 20:12:23.044224 16105 tablet_bootstrap.cc:654] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:23.044330 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:23.044587 14169 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 0.
I20240501 20:12:23.046036 14297 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 1.
I20240501 20:12:23.046139 16083 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:23.046815 16075 raft_consensus.cc:482] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:23.047004 16075 raft_consensus.cc:504] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.047351 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:23.047572 14692 raft_consensus.cc:3037] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.047957 16083 tablet_bootstrap.cc:654] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:23.048251 16075 leader_election.cc:288] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939)
I20240501 20:12:23.049022 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:23.049247 14036 raft_consensus.cc:2454] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:23.049803 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:23.050371 14692 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 1.
W20240501 20:12:23.050895 14104 leader_election.cc:341] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:23.052270 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:23.052242 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:23.052528 14692 raft_consensus.cc:2454] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:23.052522 13905 raft_consensus.cc:2454] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 0.
I20240501 20:12:23.052608 16083 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:23.053210 14105 leader_election.cc:302] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7
I20240501 20:12:23.053401 16083 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:12:23.053575 16075 raft_consensus.cc:2781] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:23.053691 16075 raft_consensus.cc:482] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:23.053778 16075 raft_consensus.cc:3037] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.054286 16083 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.054582 16083 raft_consensus.cc:374] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.054663 16083 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.054896 16083 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.055578 16083 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:23.055876 16083 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:23.056630 16075 raft_consensus.cc:504] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.058418 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:23.058511 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:23.058631 14036 raft_consensus.cc:3037] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:12:23.059059 14104 leader_election.cc:341] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:23.059352 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:23.059559 14692 raft_consensus.cc:3037] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.059989 16083 tablet_bootstrap.cc:654] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:23.061451 14036 raft_consensus.cc:2454] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:23.062331 16075 leader_election.cc:288] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Requested vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939)
I20240501 20:12:23.062695 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:23.062933 13905 raft_consensus.cc:3037] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.062187 14692 raft_consensus.cc:2454] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:23.064632 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:23.064894 14167 raft_consensus.cc:3037] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.065785 13905 raft_consensus.cc:2454] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:23.066139 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:23.066411 14036 raft_consensus.cc:3037] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.067548 14167 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 1.
I20240501 20:12:23.068322 14105 leader_election.cc:302] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7
I20240501 20:12:23.068821 13841 leader_election.cc:302] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, a5ca96c8a9a74413bff544306af10586
I20240501 20:12:23.069085 14036 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c12d6cddb7f242b6af650a10bbc8eb69 in term 1.
I20240501 20:12:23.069885 16075 raft_consensus.cc:2781] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:23.070017 16075 raft_consensus.cc:686] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Becoming Leader. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Running, Role: LEADER
I20240501 20:12:23.070665 16096 leader_election.cc:288] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [CANDIDATE]: Term 1 election: Requested vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:23.070879 16096 raft_consensus.cc:2781] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:23.071017 16096 raft_consensus.cc:686] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Becoming Leader. State: Replica: c12d6cddb7f242b6af650a10bbc8eb69, State: Running, Role: LEADER
I20240501 20:12:23.071588 16096 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.070354 16075 consensus_queue.cc:237] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.073980 16083 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:23.074215 16083 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.003s
I20240501 20:12:23.074826 14561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:23.075217 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:23.075290 16083 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.076213 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:23.076436 14298 raft_consensus.cc:3037] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.076807 16083 raft_consensus.cc:374] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.076897 16083 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.077139 14297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "0e898d8ee23441908d1593c561b368ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:23.077168 16083 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.077858 16083 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:12:23.078279 16083 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:23.079308 14298 raft_consensus.cc:2454] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e898d8ee23441908d1593c561b368ad in term 1.
I20240501 20:12:23.080968 16083 tablet_bootstrap.cc:654] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: Neither blocks nor log segments found. Creating new log.
W20240501 20:12:23.075974 14103 leader_election.cc:341] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:12:23.082890 14103 leader_election.cc:341] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:23.090499 16083 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: No bootstrap required, opened a new log
I20240501 20:12:23.090700 16083 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240501 20:12:23.091513 16083 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.091794 16083 raft_consensus.cc:374] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.091887 16083 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.091768 13765 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: term changed from 0 to 1, leader changed from <none> to c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65). New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:23.091683 13756 catalog_manager.cc:5484] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e898d8ee23441908d1593c561b368ad (127.28.136.67). New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:23.092149 16083 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.092787 16083 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:23.093791 16105 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:23.093967 16105 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.052s user 0.018s sys 0.018s
I20240501 20:12:23.095557 16105 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.095870 16105 raft_consensus.cc:374] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.096012 16105 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.096293 16105 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.097158 16105 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:12:23.097862 16105 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:23.098781 15769 raft_consensus.cc:482] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:23.098951 15769 raft_consensus.cc:504] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.100162 15769 leader_election.cc:288] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:23.100196 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:23.100430 13905 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:23.101080 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" is_pre_election: true
I20240501 20:12:23.101343 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
W20240501 20:12:23.101603 13973 leader_election.cc:341] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:12:23.101634 14298 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:23.101719 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:23.101863 16105 tablet_bootstrap.cc:654] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:23.101950 14563 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:23.102702 13972 leader_election.cc:302] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters: 3b9f58142e4a4ccd9d0cbafc97d459c7
I20240501 20:12:23.103025 15769 raft_consensus.cc:2781] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:12:23.103156 15769 raft_consensus.cc:482] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:12:23.103247 15769 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.103886 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:23.104084 14692 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:23.105382 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:23.105605 14167 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 0.
I20240501 20:12:23.106158 15769 raft_consensus.cc:504] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.107618 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:23.107825 13905 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.108503 16105 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:23.108683 16105 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240501 20:12:23.109583 16105 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.109908 16105 raft_consensus.cc:374] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.110028 16105 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.110283 16105 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.110971 16105 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:12:23.111461 16105 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Bootstrap starting.
I20240501 20:12:23.112011 15769 leader_election.cc:288] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:23.112277 13905 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:23.112373 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:23.112604 14167 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.113159 16105 tablet_bootstrap.cc:654] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:12:23.113677 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088"
I20240501 20:12:23.113893 14298 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.115218 14167 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:23.115352 16105 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7: No bootstrap required, opened a new log
I20240501 20:12:23.115495 16105 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:12:23.115950 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7"
I20240501 20:12:23.116194 14430 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.116477 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:23.116708 14692 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.117434 14298 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:23.117923 13972 leader_election.cc:302] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:23.118255 15769 raft_consensus.cc:2781] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:12:23.118841 14430 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:23.119333 14692 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:23.119378 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:23.119599 14563 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.119848 16105 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.120107 16105 raft_consensus.cc:374] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:12:23.120191 16105 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9f58142e4a4ccd9d0cbafc97d459c7, State: Initialized, Role: FOLLOWER
I20240501 20:12:23.120446 16105 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.121158 16105 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:12:23.121953 14563 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3224a319e2f4cb8a84bf9528209a9fb in term 1.
I20240501 20:12:23.125583 15769 raft_consensus.cc:686] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Becoming Leader. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Running, Role: LEADER
I20240501 20:12:23.126072 15769 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:23.130264 13756 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: term changed from 0 to 1, leader changed from <none> to c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66). New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:23.173662 15708 raft_consensus.cc:482] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:23.173895 15708 raft_consensus.cc:504] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.175226 15708 leader_election.cc:288] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:23.175801 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:23.175803 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:23.176088 14167 raft_consensus.cc:2379] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in current term 1: Already voted for candidate c12d6cddb7f242b6af650a10bbc8eb69 in this term.
I20240501 20:12:23.176304 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:23.176452 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:23.176561 14692 raft_consensus.cc:2379] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in current term 1: Already voted for candidate c12d6cddb7f242b6af650a10bbc8eb69 in this term.
I20240501 20:12:23.176725 14036 raft_consensus.cc:2379] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in current term 1: Already voted for candidate c12d6cddb7f242b6af650a10bbc8eb69 in this term.
I20240501 20:12:23.177300 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:23.177430 14367 leader_election.cc:302] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7; no voters: 0e898d8ee23441908d1593c561b368ad, 3c0f925474374c18a6cd80cff94d4295, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:23.177525 14563 raft_consensus.cc:2454] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:23.177549 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1840c628fc114b309da99d3c8a91656a" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:23.177798 14298 raft_consensus.cc:2379] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in current term 1: Already voted for candidate c12d6cddb7f242b6af650a10bbc8eb69 in this term.
I20240501 20:12:23.177909 16076 raft_consensus.cc:3037] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.180969 16076 raft_consensus.cc:2726] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:23.420847 16076 raft_consensus.cc:482] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:12:23.421133 16076 raft_consensus.cc:504] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:23.422456 16076 leader_election.cc:288] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:23.423190 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca96c8a9a74413bff544306af10586" is_pre_election: true
I20240501 20:12:23.423300 14036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" is_pre_election: true
I20240501 20:12:23.423277 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" is_pre_election: true
I20240501 20:12:23.423422 14563 raft_consensus.cc:2454] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in term 0.
I20240501 20:12:23.423566 13905 raft_consensus.cc:2379] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in current term 1: Already voted for candidate 0e898d8ee23441908d1593c561b368ad in this term.
I20240501 20:12:23.423557 14036 raft_consensus.cc:2379] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in current term 1: Already voted for candidate 0e898d8ee23441908d1593c561b368ad in this term.
I20240501 20:12:23.424238 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0f925474374c18a6cd80cff94d4295" is_pre_election: true
I20240501 20:12:23.424314 14298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692bb14bda8046b9827f1091b0946088" is_pre_election: true
I20240501 20:12:23.424505 14692 raft_consensus.cc:2379] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in current term 1: Already voted for candidate 0e898d8ee23441908d1593c561b368ad in this term.
I20240501 20:12:23.424574 14298 raft_consensus.cc:2379] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3b9f58142e4a4ccd9d0cbafc97d459c7 in current term 1: Already voted for candidate 0e898d8ee23441908d1593c561b368ad in this term.
I20240501 20:12:23.424818 14167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3808b45349de4959b929a99735e5db53" candidate_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e898d8ee23441908d1593c561b368ad" is_pre_election: true
I20240501 20:12:23.425143 14365 leader_election.cc:302] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 3b9f58142e4a4ccd9d0cbafc97d459c7, a5ca96c8a9a74413bff544306af10586; no voters: 3c0f925474374c18a6cd80cff94d4295, 692bb14bda8046b9827f1091b0946088, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:23.425488 16076 raft_consensus.cc:3037] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.428272 16076 raft_consensus.cc:2726] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:12:23.475457 16075 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.498045 16090 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.506057 16096 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.513702 15769 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.528429 15769 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:23.528580 16124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:23.529000 16124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:12:23.535156 15769 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.536984 14563 raft_consensus.cc:3037] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:12:23.538823 16124 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:12:23.540669 16096 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.557440 16121 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.557801 16120 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.562302 16120 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240501 20:12:23.624670 16075 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.630841 16090 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.650239 16096 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.665159 14563 raft_consensus.cc:3037] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:12:23.669330 16096 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.675429 16075 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.684422 16166 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:12:23.695902 16174 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:12:23.733491 14102 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:44849, user_credentials={real_user=slave}} blocked reactor thread for 54917.2us
W20240501 20:12:23.736536 14104 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:43621, user_credentials={real_user=slave}} blocked reactor thread for 58668.5us
I20240501 20:12:23.748270 16174 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 509.524 ms
time per row: 0.254762 ms
I20240501 20:12:24.258330 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.72:39739
--local_ip_for_outbound_sockets=127.28.136.72
--tserver_master_addrs=127.28.136.126:34579
--webserver_port=46005
--webserver_interface=127.28.136.72
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:12:24.638446 16221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:24.638732 16221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:24.638844 16221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:24.647074 16221 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:24.647223 16221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:24.647408 16221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.72
I20240501 20:12:24.657907 16221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.72:39739
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.28.136.72
--webserver_port=46005
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.72
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:24.659519 16221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:24.661491 16221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:24.675166 16228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:24.675918 16227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:24.676322 16221 server_base.cc:1034] running on GCE node
W20240501 20:12:24.676600 16230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:24.677258 16221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:24.678452 16221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:24.679672 16221 hybrid_clock.cc:648] HybridClock initialized: now 1714594344679608 us; error 51 us; skew 500 ppm
I20240501 20:12:24.679996 16221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:24.686661 16221 webserver.cc:460] Webserver started at http://127.28.136.72:46005/ using document root <none> and password file <none>
I20240501 20:12:24.687633 16221 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:24.687764 16221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:24.693320 16221 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 20:12:24.696512 16236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:24.698033 16221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:24.698199 16221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "aeb70769b705493bb5196527311b142b"
format_stamp: "Formatted at 2024-05-01 20:12:07 on dist-test-slave-sx0n"
I20240501 20:12:24.699004 16221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:24.722074 16221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:24.723881 16221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:24.724197 16221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:24.725409 16221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:24.727522 16221 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:24.727613 16221 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:24.727726 16221 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:24.727778 16221 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:24.781450 16221 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.72:39739
I20240501 20:12:24.781491 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.72:39739 every 8 connection(s)
I20240501 20:12:24.782894 16221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:12:24.790908 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 16221
I20240501 20:12:24.791707 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.74:45465
--local_ip_for_outbound_sockets=127.28.136.74
--tserver_master_addrs=127.28.136.126:34579
--webserver_port=42735
--webserver_interface=127.28.136.74
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:24.799844 16349 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:24.800184 16349 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:24.801187 16349 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:24.802835 13759 ts_manager.cc:194] Re-registered known tserver with Master: aeb70769b705493bb5196527311b142b (127.28.136.72:39739)
I20240501 20:12:24.803779 13759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.72:41257
W20240501 20:12:25.174161 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:25.174449 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:25.174580 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:25.183215 16353 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:25.183348 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:25.183504 16353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.74
I20240501 20:12:25.193439 16353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.74:45465
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.28.136.74
--webserver_port=42735
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.74
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:25.194872 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:25.196805 16353 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:25.209609 16359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:25.210072 16360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:25.210237 16362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:25.211867 16353 server_base.cc:1034] running on GCE node
I20240501 20:12:25.212481 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:25.213542 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:25.214763 16353 hybrid_clock.cc:648] HybridClock initialized: now 1714594345214713 us; error 35 us; skew 500 ppm
I20240501 20:12:25.215081 16353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:25.217352 16353 webserver.cc:460] Webserver started at http://127.28.136.74:42735/ using document root <none> and password file <none>
I20240501 20:12:25.218122 16353 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:25.218259 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:25.224346 16353 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:12:25.228646 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:25.230280 16353 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240501 20:12:25.230463 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "1899220968384feebca3562bd7741a21"
format_stamp: "Formatted at 2024-05-01 20:12:08 on dist-test-slave-sx0n"
I20240501 20:12:25.231597 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:25.249796 16353 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:25.250886 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:25.251236 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:25.252434 16353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:25.254020 16353 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:25.254098 16353 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:25.254207 16353 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:25.254253 16353 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:25.290946 16353 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.74:45465
I20240501 20:12:25.291025 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.74:45465 every 8 connection(s)
I20240501 20:12:25.292495 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240501 20:12:25.298228 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 16353
I20240501 20:12:25.298854 29217 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
/tmp/dist-test-taskzuvgUY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.136.73:39773
--local_ip_for_outbound_sockets=127.28.136.73
--tserver_master_addrs=127.28.136.126:34579
--webserver_port=45595
--webserver_interface=127.28.136.73
--builtin_ntp_servers=127.28.136.84:36231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:12:25.306401 16481 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:25.306702 16481 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:25.307519 16481 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:25.308950 13759 ts_manager.cc:194] Re-registered known tserver with Master: 1899220968384feebca3562bd7741a21 (127.28.136.74:45465)
I20240501 20:12:25.309752 13759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.74:59641
W20240501 20:12:25.673954 16484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:12:25.674328 16484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:12:25.674481 16484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:12:25.683449 16484 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:12:25.683579 16484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:12:25.683727 16484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.136.73
I20240501 20:12:25.694296 16484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.136.84:36231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.136.73:39773
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.28.136.73
--webserver_port=45595
--tserver_master_addrs=127.28.136.126:34579
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.136.73
--log_dir=/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:12:25.695853 16484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:12:25.698057 16484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:12:25.711098 16492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:25.711534 16493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:12:25.711952 16495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:12:25.712426 16484 server_base.cc:1034] running on GCE node
I20240501 20:12:25.713112 16484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:12:25.714051 16484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:12:25.715246 16484 hybrid_clock.cc:648] HybridClock initialized: now 1714594345715179 us; error 55 us; skew 500 ppm
I20240501 20:12:25.715600 16484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:12:25.718024 16484 webserver.cc:460] Webserver started at http://127.28.136.73:45595/ using document root <none> and password file <none>
I20240501 20:12:25.718900 16484 fs_manager.cc:359] Metadata directory not provided
I20240501 20:12:25.719050 16484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:12:25.725216 16484 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 20:12:25.728413 16501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:25.729924 16484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:25.730078 16484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "2a31af84f57c4ef18f0d98ff044229b6"
format_stamp: "Formatted at 2024-05-01 20:12:08 on dist-test-slave-sx0n"
I20240501 20:12:25.731137 16484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:12:25.768712 16484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:12:25.769754 16484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:12:25.770020 16484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:12:25.770949 16484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:12:25.772404 16484 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:12:25.772475 16484 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:25.772591 16484 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:12:25.772641 16484 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:12:25.806655 16349 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
I20240501 20:12:25.816190 16484 rpc_server.cc:307] RPC server started. Bound to: 127.28.136.73:39773
I20240501 20:12:25.816242 16613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.136.73:39773 every 8 connection(s)
I20240501 20:12:25.817771 16484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240501 20:12:25.820891 29217 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzuvgUY/build/asan/bin/kudu as pid 16484
I20240501 20:12:25.830776 16614 heartbeater.cc:347] Connected to a master server at 127.28.136.126:34579
I20240501 20:12:25.831137 16614 heartbeater.cc:464] Registering TS with master...
I20240501 20:12:25.832134 16614 heartbeater.cc:511] Master 127.28.136.126:34579 requested a full tablet report, sending...
I20240501 20:12:25.837868 13759 ts_manager.cc:194] Re-registered known tserver with Master: 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773)
I20240501 20:12:25.838735 13759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.136.73:36143
I20240501 20:12:26.312676 16481 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
I20240501 20:12:26.426924 14147 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:26.430938 16549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.442073 16416 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.461910 14674 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:26.493614 14542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:26.505275 14279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.513772 14410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:26.604398 16284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.607133 13884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:26.640158 14017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:26.841722 16614 heartbeater.cc:503] Master 127.28.136.126:34579 was elected leader, sending a full tablet report...
I20240501 20:12:26.888554 14147 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:26.890410 16416 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.916424 14674 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:26.916786 14410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:26.928299 14279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.950950 16549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:26.962603 14542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:26.967152 16284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:27.006742 13884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:27.015570 14017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:27.260648 14147 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:27.262929 16416 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:27.265199 16549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:27.296748 14279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:27.299309 14542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:27.311604 14410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:27.338964 13884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:27.341024 16284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:27.381016 14674 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:27.408640 14017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:27.516551 14429 consensus_queue.cc:237] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:27.519776 14169 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.519920 14036 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.520074 13905 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.520596 16731 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.521755 16731 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.522763 16763 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.522969 14300 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.523526 14561 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.524112 16764 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.524889 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.536948 14694 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.537552 16766 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.539239 16766 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.539783 14561 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.540823 14169 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.546967 14036 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.547678 13762 catalog_manager.cc:5484] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.551054 14297 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.557243 14692 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.566033 14429 consensus_queue.cc:237] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:27.570479 14297 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.571202 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.573410 14036 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.574102 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.574398 14563 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.575548 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.578457 16765 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:27.579237 14300 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:27.582312 14169 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.583034 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.583475 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 39bedd9f3d3c4ee2890206cb43591675 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:27.585319 14036 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:27.587059 14563 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:27.592787 14692 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.592703 13760 catalog_manager.cc:5484] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:27.591835 13905 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.596426 14169 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:27.599049 13906 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.600433 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.602034 13906 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:27.605705 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.614291 14692 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
W20240501 20:12:27.619930 14366 consensus_peers.cc:485] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39bedd9f3d3c4ee2890206cb43591675. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:27.622941 14692 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:27.625860 14561 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.626210 14300 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.626391 14429 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.626504 14036 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.626650 16780 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.626996 16620 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.627300 16099 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.628087 16786 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.628636 14169 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.630096 13905 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.630656 16786 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.630995 16099 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.634537 16099 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.642627 14169 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.643464 14036 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.649219 13757 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:27.649096 14429 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.651054 14300 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.651573 14561 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.655014 13905 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.667618 14692 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:27.678231 14169 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.679006 13905 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.680490 16794 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.681001 16099 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.683310 14429 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.684036 16099 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.685256 14561 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.685312 14297 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.686903 16099 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.687672 16786 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.688127 16794 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.691133 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet af0f6a4c2e9c43509502a242b596d9da with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:27.692201 14169 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.692800 14297 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.698716 13763 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.699669 14429 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.678014 14036 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:12:27.704120 14630 consensus_peers.cc:485] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af0f6a4c2e9c43509502a242b596d9da. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:27.704802 16794 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.707820 13905 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.708055 14561 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.709188 14036 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.717164 14037 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:27.720786 14169 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.721055 14692 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.721930 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.722177 14429 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.722268 14561 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.722522 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.723062 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.723213 13906 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.724328 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.729852 15769 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.731266 14297 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.731983 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.734613 15769 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.735283 14297 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.741505 13906 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.745061 13759 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.769600 14429 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.779793 14169 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.781026 14694 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.797842 14561 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:27.802160 14037 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:27.807504 14297 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.808452 14429 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.808858 16811 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.809543 16811 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.809638 14561 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.810303 16809 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.815198 14692 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.816565 13906 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.817528 15769 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.818109 15769 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.821244 14169 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.821038 16809 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:27.821954 13906 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:27.824585 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.825284 14430 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
W20240501 20:12:27.826735 13974 consensus_peers.cc:485] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8892d6dce05140babdeeb6c90c366b3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:27.828328 13757 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:27.828982 14692 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:27.829119 14694 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:27.832088 14297 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.832443 14561 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.832911 16099 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.833290 14429 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.833946 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.834554 14167 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.836266 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.832706 14169 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:27.837344 14037 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.838021 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.838487 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.838907 13906 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.839673 16787 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.862334 13749 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8892d6dce05140babdeeb6c90c366b3b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:27.863077 14563 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:27.874279 14297 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:27.875559 16099 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.882045 14563 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.881302 14297 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.885206 14430 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.888988 13757 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.889133 14036 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.895478 14169 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.901810 14694 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
W20240501 20:12:27.905529 14630 consensus_peers.cc:485] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e21fb0e6bac46fbbd29bbd2f1bf43b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:27.906453 13905 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.907030 16794 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.908362 14563 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.908190 13905 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.908874 16780 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.909734 14430 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.910777 14297 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.910352 16099 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.911284 16780 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.914939 14169 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.915370 14037 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:27.915884 16780 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.916337 16780 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:27.917410 16780 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.918758 14300 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.919059 13906 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.929770 14169 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.934958 13763 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.941877 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:27.943019 14430 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.943359 14561 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.944389 14037 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:27.960703 14430 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:27.963351 14561 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.963336 14694 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.963930 16486 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.964438 14037 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.964511 13906 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.964491 16486 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.964586 14167 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.965534 16731 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.965823 14297 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:27.965987 16731 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.966444 16731 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.967530 16731 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:27.970855 16731 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.971472 13906 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.971928 14561 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.978009 14694 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.982899 13762 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.983950 14300 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.993068 14037 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:27.993746 14169 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.001346 14430 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
W20240501 20:12:28.004611 14366 consensus_peers.cc:485] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6213df9f24a4e63aec805f7dfacc75d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:28.004681 13906 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.005278 16764 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.005534 14037 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.006147 16763 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.006886 14563 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.007982 14300 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.008141 14694 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.008579 16731 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.009122 16765 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.009668 16731 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.012431 16731 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:28.013778 14169 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.014312 16486 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.016377 14169 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:28.016425 14300 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:28.017603 13906 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:28.022313 13757 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:28.024135 14563 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:28.024857 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6213df9f24a4e63aec805f7dfacc75d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:28.026073 14037 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:28.026949 14692 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:28.037350 14563 consensus_queue.cc:237] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:28.040186 14300 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.040257 14692 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.041066 16825 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.041553 16631 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.042716 13906 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.042876 14430 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.043277 16825 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.043536 16845 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.049569 14169 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.050750 16846 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.058593 16631 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.059130 13906 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.060293 14037 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.060824 16845 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.064910 14429 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.069168 14694 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.070459 13763 catalog_manager.cc:5484] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.072014 14300 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.072569 14037 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.088351 14167 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.097651 16862 ts_tablet_manager.cc:902] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:28.102782 16861 ts_tablet_manager.cc:902] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:28.103998 14037 consensus_queue.cc:237] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:28.109354 14561 consensus_queue.cc:237] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:28.111171 16862 tablet_copy_client.cc:308] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:28.113247 14036 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.113809 13906 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.115053 16825 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:12:28.115340 14430 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.116344 16861 tablet_copy_client.cc:308] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:32991
I20240501 20:12:28.116699 14694 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.117265 13906 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.117309 16809 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.117815 16809 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.118186 14167 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.119073 14694 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.119130 14169 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.119661 16619 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.121837 14300 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.129205 16855 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.129528 16825 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.130067 16856 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.130360 16853 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.131088 16631 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.136368 14430 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.136704 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet af0f6a4c2e9c43509502a242b596d9da from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:45703)
I20240501 20:12:28.137065 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet af0f6a4c2e9c43509502a242b596d9da from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:45703: session id = 2a31af84f57c4ef18f0d98ff044229b6-af0f6a4c2e9c43509502a242b596d9da
I20240501 20:12:28.137198 14297 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.138362 14563 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.138512 16808 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.139158 15769 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.139426 16619 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.141579 14714 tablet_copy_source_session.cc:205] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:28.141291 16825 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.143823 14037 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.145726 13750 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6335cd6db4474648bdf39486b7c9bb09 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:28.146111 14300 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.147934 14692 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.149380 13763 catalog_manager.cc:5484] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:28.151809 16862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:28.160862 14167 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.170783 16862 tablet_copy_client.cc:783] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:28.170526 16809 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:28.171099 16862 tablet_copy_client.cc:647] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:28.173597 16862 tablet_copy_client.cc:523] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:28.175809 14297 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:28.177999 16862 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:28.179996 13906 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:28.180239 14429 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:28.185972 14430 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.191797 16862 log.cc:824] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:28.194888 14169 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:28.195559 13759 catalog_manager.cc:5484] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:28.200135 14692 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:28.200163 14563 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:28.206518 14036 consensus_queue.cc:237] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:28.210834 16882 ts_tablet_manager.cc:902] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:28.211619 14692 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.212114 16808 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.215873 13906 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.220479 16619 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.222453 14169 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.223969 15769 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.225869 14430 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:12:28.225878 13974 consensus_peers.cc:485] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64ded5ed2b224d03a7ac23eb3e749f73. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:28.226413 15769 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.227511 14563 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.229962 15769 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.233217 14451 tablet_copy_service.cc:140] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Received BeginTabletCopySession request for tablet 39bedd9f3d3c4ee2890206cb43591675 from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:58191)
I20240501 20:12:28.233471 14451 tablet_copy_service.cc:161] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Beginning new tablet copy session on tablet 39bedd9f3d3c4ee2890206cb43591675 from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:58191: session id = 1899220968384feebca3562bd7741a21-39bedd9f3d3c4ee2890206cb43591675
W20240501 20:12:28.233599 14498 consensus_peers.cc:485] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6335cd6db4474648bdf39486b7c9bb09. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:28.234362 14431 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.235308 16619 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.237265 13906 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.237396 14563 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.237463 13904 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.242332 13749 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 64ded5ed2b224d03a7ac23eb3e749f73 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:28.242926 14297 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.243106 13762 catalog_manager.cc:5484] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:28.243619 16811 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.245088 14692 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.245277 14300 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.248680 14451 tablet_copy_source_session.cc:205] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:28.257683 16861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:28.277518 14169 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:28.281841 16861 tablet_copy_client.cc:783] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:28.282575 16861 tablet_copy_client.cc:647] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:28.285336 16861 tablet_copy_client.cc:523] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:28.289939 16861 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:28.319262 16861 log.cc:824] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:28.328042 16882 tablet_copy_client.cc:308] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:28.332613 16862 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:28.333324 16862 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:28.346822 16862 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.169s user 0.066s sys 0.011s
I20240501 20:12:28.352111 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet 8892d6dce05140babdeeb6c90c366b3b from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:40007)
I20240501 20:12:28.352452 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet 8892d6dce05140babdeeb6c90c366b3b from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:40007: session id = aeb70769b705493bb5196527311b142b-8892d6dce05140babdeeb6c90c366b3b
I20240501 20:12:28.355152 14036 consensus_queue.cc:237] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:28.356252 14058 tablet_copy_source_session.cc:205] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:28.362043 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:28.362776 16862 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:28.364312 16862 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:28.371018 14563 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.371703 15769 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.371909 16862 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:28.379060 14430 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.379760 15769 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.386090 16862 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.039s user 0.011s sys 0.000s
I20240501 20:12:28.387092 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-af0f6a4c2e9c43509502a242b596d9da received from {username='slave'} at 127.28.136.73:45703
I20240501 20:12:28.387269 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-af0f6a4c2e9c43509502a242b596d9da on tablet af0f6a4c2e9c43509502a242b596d9da with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:28.389869 16862 ts_tablet_manager.cc:902] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:28.390655 14692 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.390642 13904 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.391265 16808 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.393045 16811 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.394852 14169 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.395749 14300 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.396098 16811 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.396407 15769 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.409726 16882 tablet_copy_client.cc:783] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:28.410315 16882 tablet_copy_client.cc:647] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:28.413585 16882 tablet_copy_client.cc:523] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:28.419181 16811 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.427982 14692 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.430464 16861 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:28.430943 16862 tablet_copy_client.cc:308] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:28.432376 13904 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.431929 14430 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.433591 16861 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:28.435609 14300 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.436625 16861 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.147s user 0.066s sys 0.000s
I20240501 20:12:28.437829 13759 catalog_manager.cc:5484] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:28.447959 14167 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.453420 16861 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:28.454195 16861 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:28.454967 16861 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:28.455706 16882 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:28.465592 16861 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.029s user 0.004s sys 0.009s
I20240501 20:12:28.469573 16435 raft_consensus.cc:1203] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:28.472191 14036 consensus_queue.cc:237] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:28.473939 16882 log.cc:824] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:12:28.478129 14451 tablet_copy_service.cc:342] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Request end of tablet copy session 1899220968384feebca3562bd7741a21-39bedd9f3d3c4ee2890206cb43591675 received from {username='slave'} at 127.28.136.74:58191
I20240501 20:12:28.478335 14451 tablet_copy_service.cc:433] P 3b9f58142e4a4ccd9d0cbafc97d459c7: ending tablet copy session 1899220968384feebca3562bd7741a21-39bedd9f3d3c4ee2890206cb43591675 on tablet 39bedd9f3d3c4ee2890206cb43591675 with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:28.481390 16861 ts_tablet_manager.cc:902] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:28.487560 13904 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.488225 16811 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.491868 14169 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.492923 16811 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.497640 14300 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:12:28.498262 13974 consensus_peers.cc:485] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60c32a552af1499dab8884bae2287ce3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:28.498621 16811 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.501391 16811 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.505472 14300 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.510485 13749 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 60c32a552af1499dab8884bae2287ce3 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:28.514174 13759 catalog_manager.cc:5484] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.520229 14430 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.520887 16809 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.522557 14430 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.534864 14561 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.535857 16619 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.537535 14561 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.542342 14561 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.546821 16882 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:28.547422 16882 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:28.548072 16882 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.093s user 0.057s sys 0.018s
I20240501 20:12:28.553398 13904 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.554783 16882 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:28.555441 16882 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:28.556041 16882 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:28.558063 14169 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.588999 14694 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.589722 16882 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.041s user 0.010s sys 0.025s
I20240501 20:12:28.592862 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session aeb70769b705493bb5196527311b142b-8892d6dce05140babdeeb6c90c366b3b received from {username='slave'} at 127.28.136.72:40007
I20240501 20:12:28.593052 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session aeb70769b705493bb5196527311b142b-8892d6dce05140babdeeb6c90c366b3b on tablet 8892d6dce05140babdeeb6c90c366b3b with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:28.593463 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:45703)
I20240501 20:12:28.593688 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:45703: session id = 2a31af84f57c4ef18f0d98ff044229b6-4e21fb0e6bac46fbbd29bbd2f1bf43b4
I20240501 20:12:28.595942 14714 tablet_copy_source_session.cc:205] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:28.599577 16811 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.601974 16898 ts_tablet_manager.cc:902] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:28.604207 14691 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.604993 16861 tablet_copy_client.cc:308] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:32991
I20240501 20:12:28.605758 14451 tablet_copy_service.cc:140] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Received BeginTabletCopySession request for tablet c6213df9f24a4e63aec805f7dfacc75d from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:58191)
I20240501 20:12:28.605960 14451 tablet_copy_service.cc:161] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Beginning new tablet copy session on tablet c6213df9f24a4e63aec805f7dfacc75d from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:58191: session id = 1899220968384feebca3562bd7741a21-c6213df9f24a4e63aec805f7dfacc75d
I20240501 20:12:28.609092 14451 tablet_copy_source_session.cc:205] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:28.610550 16861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:28.614562 16862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:28.616401 14036 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:28.623229 16569 raft_consensus.cc:1203] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:28.624817 16862 tablet_copy_client.cc:783] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:28.625135 16862 tablet_copy_client.cc:647] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:28.626451 16861 tablet_copy_client.cc:783] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:28.626710 16861 tablet_copy_client.cc:647] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:28.627321 16862 tablet_copy_client.cc:523] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:28.628795 16861 tablet_copy_client.cc:523] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:28.630328 14563 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.630398 16862 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:28.630995 16619 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.631166 14691 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.631800 16619 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.633301 16861 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:28.637089 14300 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.637908 16619 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.648550 14429 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.649226 14169 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.649274 16619 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.649860 16619 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:28.652014 16933 ts_tablet_manager.cc:902] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:28.665760 13904 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:28.666467 16809 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:12:28.699090 16928 log.cc:924] Time spent T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20240501 20:12:28.705783 16861 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:28.706248 16861 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:28.706573 16861 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.073s user 0.050s sys 0.016s
I20240501 20:12:28.707672 16861 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:28.708218 16861 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:28.708441 16861 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:28.709172 16861 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 20:12:28.710068 14451 tablet_copy_service.cc:342] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Request end of tablet copy session 1899220968384feebca3562bd7741a21-c6213df9f24a4e63aec805f7dfacc75d received from {username='slave'} at 127.28.136.74:58191
I20240501 20:12:28.710215 14451 tablet_copy_service.cc:433] P 3b9f58142e4a4ccd9d0cbafc97d459c7: ending tablet copy session 1899220968384feebca3562bd7741a21-c6213df9f24a4e63aec805f7dfacc75d on tablet c6213df9f24a4e63aec805f7dfacc75d with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:28.725250 16619 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.730511 14563 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.730748 14169 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.735045 14430 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.736059 16898 tablet_copy_client.cc:308] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:28.739266 14691 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.740206 13763 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.753813 14297 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.777164 16862 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:28.777737 16862 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:28.778100 16862 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.148s user 0.061s sys 0.021s
I20240501 20:12:28.779340 16862 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:28.779743 16862 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:28.779997 16862 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:28.783427 16862 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:12:28.786350 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-4e21fb0e6bac46fbbd29bbd2f1bf43b4 received from {username='slave'} at 127.28.136.73:45703
I20240501 20:12:28.786476 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-4e21fb0e6bac46fbbd29bbd2f1bf43b4 on tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:28.810549 13904 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:28.815354 14036 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:28.819625 14563 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.820927 16808 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.821395 14691 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.822156 15769 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.822326 14429 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.822878 14169 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.825141 16809 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:12:28.825824 13974 consensus_peers.cc:485] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe1592bebd6b4a858987847446f15a16. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:28.826598 15769 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.827078 16304 raft_consensus.cc:1203] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:28.827243 13906 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.829808 16933 tablet_copy_client.cc:308] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:28.831864 16809 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.832307 16811 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.832453 14429 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.834815 14297 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:28.835337 16808 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:28.837034 14297 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.840790 13763 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.847859 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet 64ded5ed2b224d03a7ac23eb3e749f73 from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:45763)
I20240501 20:12:28.848065 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet 64ded5ed2b224d03a7ac23eb3e749f73 from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:45763: session id = 2a31af84f57c4ef18f0d98ff044229b6-64ded5ed2b224d03a7ac23eb3e749f73
I20240501 20:12:28.848515 14691 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.851055 14058 tablet_copy_source_session.cc:205] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:28.854084 16933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ded5ed2b224d03a7ac23eb3e749f73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:28.859273 13904 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.861857 13749 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fe1592bebd6b4a858987847446f15a16 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:28.864684 16569 raft_consensus.cc:1203] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:28.865339 16933 tablet_copy_client.cc:783] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:28.865582 16933 tablet_copy_client.cc:647] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:28.868353 14169 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.868348 14563 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:28.873059 16933 tablet_copy_client.cc:523] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:28.876366 16933 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:28.880319 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet 6335cd6db4474648bdf39486b7c9bb09 from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:37299)
I20240501 20:12:28.880546 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet 6335cd6db4474648bdf39486b7c9bb09 from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:37299: session id = 2a31af84f57c4ef18f0d98ff044229b6-6335cd6db4474648bdf39486b7c9bb09
I20240501 20:12:28.884335 14583 tablet_copy_source_session.cc:205] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:28.888160 16898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:28.894944 16898 tablet_copy_client.cc:783] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:28.895259 16898 tablet_copy_client.cc:647] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:28.897625 16898 tablet_copy_client.cc:523] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:28.901027 16898 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:28.916108 16436 raft_consensus.cc:1203] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:28.921442 16731 raft_consensus.cc:1050] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:28.922794 16731 consensus_queue.cc:237] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:28.934460 16436 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:28.935245 14169 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:28.936223 14692 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:28.936585 14297 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:28.936975 14036 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:28.937727 16766 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.937897 13904 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:28.938429 16766 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.938962 16766 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.941272 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.941820 16766 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.953039 14563 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:28.953779 16731 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.955624 16924 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:28.967809 16898 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:28.968343 16898 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:28.968649 16898 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.068s user 0.047s sys 0.005s
I20240501 20:12:28.969604 16898 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:28.969964 16898 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:28.970188 16898 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:28.970916 16898 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:12:28.971828 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-6335cd6db4474648bdf39486b7c9bb09 received from {username='slave'} at 127.28.136.73:37299
I20240501 20:12:28.972015 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-6335cd6db4474648bdf39486b7c9bb09 on tablet 6335cd6db4474648bdf39486b7c9bb09 with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:28.978456 16766 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:28.979436 14692 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:28.980392 14168 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:28.984014 13904 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:28.985764 14036 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:28.994118 13763 catalog_manager.cc:5484] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.006317 14300 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.017601 14563 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.019364 16882 ts_tablet_manager.cc:902] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:29.024618 14429 consensus_queue.cc:237] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:29.014267 16436 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.033356 14169 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.033882 14691 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.035321 14036 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.035611 13904 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.035965 16766 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.036618 16766 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.037770 16924 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.039376 16486 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.042264 16435 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.044586 16924 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.046221 16924 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.048512 14297 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.048285 14036 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.051827 13904 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.053056 16766 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.055296 13763 catalog_manager.cc:5484] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.058101 14297 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.059065 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39bedd9f3d3c4ee2890206cb43591675 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:29.060923 14691 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.061064 14169 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.063902 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 with delete_type TABLET_DATA_TOMBSTONED (TS a5ca96c8a9a74413bff544306af10586 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59144
I20240501 20:12:29.066974 16916 raft_consensus.cc:1050] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:29.068277 16916 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:29.079718 16436 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.095861 16882 tablet_copy_client.cc:308] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:29.096174 16569 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.096311 16970 tablet_replica.cc:330] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: stopping tablet replica
I20240501 20:12:29.096881 14297 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.097164 16970 raft_consensus.cc:2229] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:29.097769 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet 60c32a552af1499dab8884bae2287ce3 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:40007)
I20240501 20:12:29.097940 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet 60c32a552af1499dab8884bae2287ce3 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:40007: session id = aeb70769b705493bb5196527311b142b-60c32a552af1499dab8884bae2287ce3
I20240501 20:12:29.098203 14429 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.098783 16970 raft_consensus.cc:2258] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:29.098891 16923 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.100399 16914 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.100891 14563 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.101039 16914 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.102203 14036 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.103399 16099 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.104001 16099 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.103186 16970 ts_tablet_manager.cc:1875] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:29.114614 14169 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.115312 16914 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.117545 14058 tablet_copy_source_session.cc:205] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:29.120049 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:29.129683 16882 tablet_copy_client.cc:783] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:29.130270 16882 tablet_copy_client.cc:647] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:29.132812 16882 tablet_copy_client.cc:523] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:29.137876 13904 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.138654 16916 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.142328 16933 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:29.142817 16933 tablet_bootstrap.cc:492] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:29.143129 16933 ts_tablet_manager.cc:1367] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.267s user 0.048s sys 0.007s
I20240501 20:12:29.144076 16933 raft_consensus.cc:348] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:29.144469 16933 raft_consensus.cc:729] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:29.144687 16933 consensus_queue.cc:260] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:29.145402 16933 ts_tablet_manager.cc:1398] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:29.149320 16569 raft_consensus.cc:1203] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:29.158545 16916 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.160149 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-64ded5ed2b224d03a7ac23eb3e749f73 received from {username='slave'} at 127.28.136.73:45763
I20240501 20:12:29.159624 14561 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.160327 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-64ded5ed2b224d03a7ac23eb3e749f73 on tablet 64ded5ed2b224d03a7ac23eb3e749f73 with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:29.159117 14169 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.166047 13759 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.168294 16970 ts_tablet_manager.cc:1888] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:29.168659 16970 log.cc:1196] T 39bedd9f3d3c4ee2890206cb43591675 P a5ca96c8a9a74413bff544306af10586: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/wals/39bedd9f3d3c4ee2890206cb43591675
I20240501 20:12:29.168176 14037 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.171749 16882 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:29.173334 13750 catalog_manager.cc:4830] TS a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): tablet 39bedd9f3d3c4ee2890206cb43591675 (table rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]) successfully deleted
I20240501 20:12:29.174028 16569 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.174522 14297 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.179991 14431 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.180400 13906 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.191972 14691 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:29.195461 14169 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.195941 14563 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.196657 16974 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.196779 16569 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.197916 16780 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.198465 16780 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.199393 14037 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.199793 14429 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.200708 16780 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.202781 16974 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.203550 16786 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.205344 14563 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.206347 14037 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.208577 16569 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.209414 14169 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.210376 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af0f6a4c2e9c43509502a242b596d9da with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:29.214685 15769 raft_consensus.cc:1050] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:29.215288 13759 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:29.215917 15769 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:29.220717 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet af0f6a4c2e9c43509502a242b596d9da with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:29.223085 13905 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.228658 16099 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.230314 14429 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.230877 13904 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.237546 16991 tablet_replica.cc:330] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:29.240511 16991 raft_consensus.cc:2229] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:29.241145 16991 raft_consensus.cc:2258] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:29.243306 16991 ts_tablet_manager.cc:1875] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:29.250494 16991 ts_tablet_manager.cc:1888] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:29.250825 16991 log.cc:1196] T af0f6a4c2e9c43509502a242b596d9da P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/af0f6a4c2e9c43509502a242b596d9da
I20240501 20:12:29.252171 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet af0f6a4c2e9c43509502a242b596d9da (table rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]) successfully deleted
I20240501 20:12:29.264367 14169 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.265039 16811 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.265961 13904 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.266587 14691 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.266676 16808 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.267019 14297 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.267027 16304 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.267170 16944 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.267731 16944 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.267972 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.269757 14563 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.270398 16811 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.278237 14429 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.278888 16811 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.300428 16882 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:29.300896 16882 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:29.302860 16916 raft_consensus.cc:1050] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:29.302645 16811 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.304044 16916 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:29.309161 14169 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.312283 16882 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.141s user 0.050s sys 0.022s
I20240501 20:12:29.313484 16882 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:29.313831 16882 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:29.314057 16882 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:29.317183 16882 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:12:29.323894 16569 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.324880 13757 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.326589 16099 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.332912 13904 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.336704 16486 raft_consensus.cc:1050] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:29.337805 14300 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.337900 14561 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.338202 16486 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:29.339627 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.340274 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.339960 14692 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.340840 16304 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.343360 14431 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.343672 14168 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.344162 16099 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.344884 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session aeb70769b705493bb5196527311b142b-60c32a552af1499dab8884bae2287ce3 received from {username='slave'} at 127.28.136.72:40007
I20240501 20:12:29.345033 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session aeb70769b705493bb5196527311b142b-60c32a552af1499dab8884bae2287ce3 on tablet 60c32a552af1499dab8884bae2287ce3 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:29.347621 14563 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.349220 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.364090 14037 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:29.370385 16304 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.370602 14169 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.370863 14691 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.371147 13904 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.371146 16808 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.371490 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.371965 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.372270 16809 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.376333 14037 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.376917 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.377812 16811 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.378401 13904 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.384860 13749 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8892d6dce05140babdeeb6c90c366b3b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:29.385841 13757 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.386215 14692 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.390864 16304 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.393992 13904 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.394151 14297 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.395084 16916 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.397599 16882 ts_tablet_manager.cc:902] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:29.401217 14410 tablet_service.cc:1514] Processing DeleteTablet for tablet 8892d6dce05140babdeeb6c90c366b3b with delete_type TABLET_DATA_TOMBSTONED (TS 3b9f58142e4a4ccd9d0cbafc97d459c7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54284
I20240501 20:12:29.413688 14169 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.416936 16786 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.418306 16436 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.418270 13905 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.418992 16966 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.421986 14299 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.423221 14431 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.423933 14037 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.425130 14168 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.426242 14562 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.431942 13763 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.436072 14296 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.436506 16568 raft_consensus.cc:1203] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:29.437151 16766 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.438781 14168 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.439308 14035 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.440878 14560 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.442185 14298 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.442931 16808 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.443519 16944 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.445012 16766 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.448246 16766 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.449756 13905 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.450258 14562 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.450019 14296 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.450098 14560 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:29.456596 16924 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.457263 16966 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.458647 14692 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.459182 16966 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.485433 16966 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.492698 14560 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.492908 14168 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.493422 14296 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.494388 13758 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:29.491184 16436 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.500893 13906 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.509244 14691 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.511708 17015 tablet_replica.cc:330] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: stopping tablet replica
I20240501 20:12:29.512344 17015 raft_consensus.cc:2229] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 20:12:29.519661 17001 log.cc:924] Time spent T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Append to log took a long time: real 0.198s user 0.000s sys 0.000s
I20240501 20:12:29.520145 17015 pending_rounds.cc:62] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: Trying to abort 1 pending ops.
I20240501 20:12:29.520406 17015 pending_rounds.cc:68] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022978454384959488 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8892d6dce05140babdeeb6c90c366b3b" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } } }
I20240501 20:12:29.521250 16569 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.523713 17015 raft_consensus.cc:2894] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 20:12:29.523939 17015 raft_consensus.cc:2866] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:12:29.524137 17015 raft_consensus.cc:2258] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:29.524539 14037 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.526405 14692 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:29.530192 14295 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.530488 14168 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.530900 16786 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.532200 14037 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.532727 13905 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.532727 16916 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.532863 16569 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.534849 16916 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.536042 16780 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.531255 14431 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.536790 16780 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.537582 14429 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:29.539021 16916 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.541965 14560 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.542574 16486 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.542709 16436 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.544016 16099 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.544874 16966 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.544409 14037 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.550316 13905 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.551914 16966 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.553834 14692 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.553634 14431 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.553504 16569 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.555069 16924 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.557277 14169 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.558421 14168 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.559149 16966 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.561833 16924 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.562712 13905 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.564239 14036 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.566414 16731 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.567957 13764 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.569017 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:29.569494 16983 raft_consensus.cc:1050] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:29.570470 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6213df9f24a4e63aec805f7dfacc75d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:29.570950 16983 consensus_queue.cc:237] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:29.570910 14169 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.576090 14691 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.579988 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with delete_type TABLET_DATA_TOMBSTONED (TS a5ca96c8a9a74413bff544306af10586 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59144
I20240501 20:12:29.580397 16970 tablet_replica.cc:330] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: stopping tablet replica
I20240501 20:12:29.580648 16970 raft_consensus.cc:2229] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:29.580930 14036 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.581985 16970 raft_consensus.cc:2258] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:29.582821 17015 ts_tablet_manager.cc:1875] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:29.583125 16970 ts_tablet_manager.cc:1875] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:29.586748 14295 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.588163 13757 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.590044 16970 ts_tablet_manager.cc:1888] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:29.590248 16970 log.cc:1196] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/wals/4e21fb0e6bac46fbbd29bbd2f1bf43b4
I20240501 20:12:29.591923 13750 catalog_manager.cc:4830] TS a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (table rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]) successfully deleted
I20240501 20:12:29.593432 16569 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.594008 16631 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.594944 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet c6213df9f24a4e63aec805f7dfacc75d with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:29.579293 13904 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.595351 16991 tablet_replica.cc:330] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:29.595551 16991 raft_consensus.cc:2229] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:29.595917 16991 raft_consensus.cc:2258] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:29.597362 16991 ts_tablet_manager.cc:1875] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:29.598199 16304 raft_consensus.cc:1203] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:29.603927 13905 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.604389 14431 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.604799 14037 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.604665 16631 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.605170 16853 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.605811 16853 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.610450 14295 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.611614 16631 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.602247 17015 ts_tablet_manager.cc:1888] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:29.620446 17015 log.cc:1196] T 8892d6dce05140babdeeb6c90c366b3b P 3b9f58142e4a4ccd9d0cbafc97d459c7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/wals/8892d6dce05140babdeeb6c90c366b3b
I20240501 20:12:29.622660 16882 tablet_copy_client.cc:308] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:29.623467 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet fe1592bebd6b4a858987847446f15a16 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:40007)
I20240501 20:12:29.623664 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet fe1592bebd6b4a858987847446f15a16 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:40007: session id = aeb70769b705493bb5196527311b142b-fe1592bebd6b4a858987847446f15a16
I20240501 20:12:29.626518 14058 tablet_copy_source_session.cc:205] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:29.626230 13751 catalog_manager.cc:4830] TS 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): tablet 8892d6dce05140babdeeb6c90c366b3b (table rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]) successfully deleted
I20240501 20:12:29.634539 16991 ts_tablet_manager.cc:1888] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:29.634804 16991 log.cc:1196] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/c6213df9f24a4e63aec805f7dfacc75d
I20240501 20:12:29.635973 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet c6213df9f24a4e63aec805f7dfacc75d (table rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]) successfully deleted
I20240501 20:12:29.636126 16882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:29.637058 14169 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.637938 17032 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.639698 14692 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.640285 17032 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.642549 16882 tablet_copy_client.cc:783] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:29.642863 16882 tablet_copy_client.cc:647] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
W20240501 20:12:29.645161 17010 log.cc:924] Time spent T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240501 20:12:29.646142 16882 tablet_copy_client.cc:523] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:29.650141 16436 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.651774 16853 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.653429 14560 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:29.653347 14296 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.658579 16568 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.658790 13905 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.661660 13762 catalog_manager.cc:5484] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.665140 14431 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.673250 14692 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.675832 14036 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.682246 16882 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:29.680833 14169 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.697513 14560 consensus_queue.cc:237] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:29.702653 16568 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.703449 14692 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.703567 14296 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.704270 17042 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.704653 16983 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.705339 16853 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.708230 13905 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.709486 14169 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.709584 17042 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.710631 17042 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.712000 14036 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.712754 16853 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.714414 17032 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.717938 16568 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.718010 14169 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.726055 13757 catalog_manager.cc:5484] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.728916 13905 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.733695 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6335cd6db4474648bdf39486b7c9bb09 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:29.734855 14692 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.736335 14296 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.738160 14036 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.741071 14410 tablet_service.cc:1514] Processing DeleteTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 with delete_type TABLET_DATA_TOMBSTONED (TS 3b9f58142e4a4ccd9d0cbafc97d459c7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54284
I20240501 20:12:29.741822 17015 tablet_replica.cc:330] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: stopping tablet replica
I20240501 20:12:29.742025 17015 raft_consensus.cc:2229] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:29.742482 17015 raft_consensus.cc:2258] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:29.747817 16882 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:29.748064 17015 ts_tablet_manager.cc:1875] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:29.748345 16882 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:29.749020 16882 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.067s user 0.035s sys 0.016s
I20240501 20:12:29.749819 16882 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:29.750133 16882 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:29.750382 16882 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:29.751091 16882 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:12:29.752168 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session aeb70769b705493bb5196527311b142b-fe1592bebd6b4a858987847446f15a16 received from {username='slave'} at 127.28.136.72:40007
I20240501 20:12:29.752305 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session aeb70769b705493bb5196527311b142b-fe1592bebd6b4a858987847446f15a16 on tablet fe1592bebd6b4a858987847446f15a16 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:29.759905 17015 ts_tablet_manager.cc:1888] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:29.760020 17015 log.cc:1196] T 6335cd6db4474648bdf39486b7c9bb09 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/wals/6335cd6db4474648bdf39486b7c9bb09
I20240501 20:12:29.760710 13751 catalog_manager.cc:4830] TS 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): tablet 6335cd6db4474648bdf39486b7c9bb09 (table rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]) successfully deleted
I20240501 20:12:29.868314 15769 raft_consensus.cc:1050] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:29.869716 15769 consensus_queue.cc:237] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:29.882218 13905 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.882246 14169 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.882903 16619 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.883212 16811 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.884263 14692 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.885071 16568 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.885597 16811 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.886147 16944 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.888813 14296 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.889214 14431 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.889385 14562 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:29.889415 16944 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.889905 16944 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.890779 16811 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:29.891160 16619 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.893213 16568 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.896411 14169 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.897020 13905 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.897039 14692 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.901401 13757 catalog_manager.cc:5484] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.913589 14562 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.914088 14296 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.917552 14431 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.920940 14035 consensus_queue.cc:237] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:29.923466 14169 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.923480 13905 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.923722 14296 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.924129 15769 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.924643 14429 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.924651 15769 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.924949 16808 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.926720 16619 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.927214 16568 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.927695 14692 raft_consensus.cc:1261] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:29.928144 16944 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.928720 15769 consensus_queue.cc:1035] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:29.930330 15769 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.931221 16568 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.935443 13749 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 64ded5ed2b224d03a7ac23eb3e749f73 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:29.934288 14169 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.935557 14431 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.935863 13905 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.937004 14692 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.941557 13763 catalog_manager.cc:5484] T 64ded5ed2b224d03a7ac23eb3e749f73 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.945072 14296 raft_consensus.cc:2932] T 64ded5ed2b224d03a7ac23eb3e749f73 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:29.945873 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 64ded5ed2b224d03a7ac23eb3e749f73 with delete_type TABLET_DATA_TOMBSTONED (TS a5ca96c8a9a74413bff544306af10586 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59144
I20240501 20:12:29.946306 16970 tablet_replica.cc:330] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: stopping tablet replica
I20240501 20:12:29.946529 16970 raft_consensus.cc:2229] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:29.946868 16970 raft_consensus.cc:2258] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:29.948341 16970 ts_tablet_manager.cc:1875] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:29.953790 16970 ts_tablet_manager.cc:1888] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:29.953905 16970 log.cc:1196] T 64ded5ed2b224d03a7ac23eb3e749f73 P a5ca96c8a9a74413bff544306af10586: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/wals/64ded5ed2b224d03a7ac23eb3e749f73
I20240501 20:12:29.954581 13750 catalog_manager.cc:4830] TS a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): tablet 64ded5ed2b224d03a7ac23eb3e749f73 (table rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]) successfully deleted
I20240501 20:12:29.959553 16304 raft_consensus.cc:1203] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:30.029714 13905 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:30.032835 14692 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.033480 14169 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.034108 17081 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.034303 14296 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.034478 16096 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.035012 16096 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.035259 14035 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.036957 14431 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.040365 16096 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.040882 16096 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.047744 14562 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.048255 17081 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.048861 16096 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.050725 14035 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.056556 13759 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:30.056527 14296 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.056998 14429 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.074282 14562 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.075168 14169 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.077414 13905 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.080757 14562 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.081329 17081 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.082537 14296 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.082624 14035 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.083400 17095 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.083788 17095 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.088618 17082 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.088990 14035 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.097000 14429 consensus_queue.cc:237] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:30.097060 13758 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.096909 14691 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.099736 14169 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.099845 13905 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.101146 16966 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.101224 13752 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1840c628fc114b309da99d3c8a91656a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:30.102468 14296 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.102806 14562 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.107569 16966 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.108345 14431 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.109315 14168 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.110090 17082 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.110920 14035 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.110950 14562 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.111205 14694 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.111542 17082 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.111590 16486 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.111841 16766 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.113032 14431 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.117173 14692 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.118954 14296 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.119046 16731 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.119910 14168 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.120411 16731 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.124588 16486 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.125545 17081 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.125488 14562 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.127571 14692 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.129534 14169 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.130092 13905 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
W20240501 20:12:30.136766 13841 consensus_peers.cc:485] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1840c628fc114b309da99d3c8a91656a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:30.137904 13757 catalog_manager.cc:5484] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.140444 14036 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.147486 14431 consensus_queue.cc:237] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.150972 14036 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:12:30.151108 14366 consensus_peers.cc:485] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ae91bfd01694bea81cda5ffc9e7bc77. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:30.151309 13905 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.151842 16486 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.152000 14560 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.152606 16486 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.153118 16486 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.155741 14169 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.155949 16486 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.157321 17027 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.158913 14295 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.158947 14692 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.159776 17027 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.160957 17027 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.161460 14692 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.162191 14298 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.162714 14169 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.165880 14692 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.168015 14298 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.169371 13757 catalog_manager.cc:5484] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.156666 14036 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.179442 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:30.181342 14560 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.182168 13905 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.200850 14694 consensus_queue.cc:237] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:30.203691 14431 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.203847 14036 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.204387 14296 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.204502 16780 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.205005 16786 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.205327 16099 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.205499 14560 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.207208 14169 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.207260 16099 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.207774 13905 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.207891 16099 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.208369 16099 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.218071 17112 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.218732 14169 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.226540 13758 catalog_manager.cc:5484] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.231235 16619 raft_consensus.cc:1050] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:30.231698 13904 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.232483 16619 consensus_queue.cc:237] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:30.233855 14036 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.237735 14560 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.243222 14694 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.243786 15769 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.246040 14560 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.246109 14431 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.246662 15769 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.250162 14296 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.250855 14429 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.251564 16304 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.252054 16811 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.253094 16944 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.254865 13904 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.254994 14169 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.255846 14298 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.258021 16944 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.258572 16944 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.259459 16811 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.261610 14694 consensus_queue.cc:237] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.265974 13905 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.266618 16786 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.270798 14036 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.271417 16780 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.287317 16811 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.288885 14296 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.289637 14563 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.289824 16780 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.289474 14562 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.290110 16099 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.299649 13758 catalog_manager.cc:5484] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.301440 14169 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.302691 14563 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.305084 14429 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.307638 17126 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.312958 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 56c262e138874431a07f6d0bfcb516bf with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:30.313541 14296 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.313627 14694 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.315629 13905 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.318403 14036 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.319166 13764 catalog_manager.cc:5484] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:30.350905 14429 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.352407 14169 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.355326 13904 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.358739 14298 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.363703 17112 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.367480 16304 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.369372 14036 consensus_queue.cc:237] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:30.375943 14694 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.376534 16944 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.376562 14169 consensus_queue.cc:237] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:30.381467 14168 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.381708 14429 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.381948 17112 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.382472 15769 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.382771 16808 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.383265 13904 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.385303 16304 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.386862 15769 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.387390 15769 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.389134 14560 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.389910 16944 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.391415 16811 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.392076 14429 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.396920 13749 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60c32a552af1499dab8884bae2287ce3 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:30.396966 14694 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.398298 14168 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.401520 14431 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
W20240501 20:12:30.402037 14629 consensus_peers.cc:485] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56c262e138874431a07f6d0bfcb516bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:30.401604 14166 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.407979 13758 catalog_manager.cc:5484] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.400316 14560 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.409564 13904 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.419773 16304 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.425488 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c32a552af1499dab8884bae2287ce3 with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:30.430521 15769 raft_consensus.cc:1050] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:30.430833 17136 tablet_replica.cc:330] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:30.431141 17136 raft_consensus.cc:2229] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:30.431496 17136 raft_consensus.cc:2258] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:30.431814 15769 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:30.432668 17136 ts_tablet_manager.cc:1875] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:30.438364 17136 ts_tablet_manager.cc:1888] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:30.438544 17136 log.cc:1196] T 60c32a552af1499dab8884bae2287ce3 P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/60c32a552af1499dab8884bae2287ce3
I20240501 20:12:30.439714 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet 60c32a552af1499dab8884bae2287ce3 (table rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]) successfully deleted
I20240501 20:12:30.442133 14036 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.442750 16166 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.444866 13904 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.445444 16166 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.447538 14694 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.447937 14560 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.448272 16811 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.448573 16944 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.458652 13904 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.458901 16304 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.459592 15769 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.460011 16619 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.460916 14429 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.461555 16619 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.467423 14298 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.467206 16619 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.470127 14560 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.473879 14562 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.474341 14298 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.474615 16166 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.475458 13762 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:30.475525 14429 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.476899 16166 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.481343 14694 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.484869 16808 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.488696 16304 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.489811 16166 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.493503 14563 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.499750 13757 catalog_manager.cc:5484] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.507817 14692 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.508046 14429 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.473313 13904 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.512295 14169 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:30.515381 15769 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:30.516927 14298 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.523977 17149 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.524812 17149 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.526263 14298 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.532425 14429 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.532860 14036 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.533676 14694 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.534198 13904 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:30.525897 14169 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.541122 17154 ts_tablet_manager.cc:902] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939)
I20240501 20:12:30.544643 14166 consensus_queue.cc:237] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.550359 14562 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.550489 14298 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.551131 17149 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.551329 14036 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.551601 17149 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.552045 17149 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.557924 13904 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.558490 16166 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.559123 14036 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.561514 17142 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240501 20:12:30.562706 14694 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.562739 14429 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.563519 17148 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.563246 13904 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.563895 17149 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.565425 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3808b45349de4959b929a99735e5db53 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:30.565203 14429 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.567458 14562 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.567692 14298 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:30.571108 13757 catalog_manager.cc:5484] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:30.571604 14694 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
W20240501 20:12:30.594524 14104 consensus_peers.cc:485] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3808b45349de4959b929a99735e5db53. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:30.599517 17154 tablet_copy_client.cc:308] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:42939
I20240501 20:12:30.604192 14694 consensus_queue.cc:237] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:30.606559 13926 tablet_copy_service.cc:140] P c12d6cddb7f242b6af650a10bbc8eb69: Received BeginTabletCopySession request for tablet 1840c628fc114b309da99d3c8a91656a from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:52915)
I20240501 20:12:30.606827 13926 tablet_copy_service.cc:161] P c12d6cddb7f242b6af650a10bbc8eb69: Beginning new tablet copy session on tablet 1840c628fc114b309da99d3c8a91656a from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:52915: session id = 1899220968384feebca3562bd7741a21-1840c628fc114b309da99d3c8a91656a
I20240501 20:12:30.607156 14562 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.607652 14298 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.607988 14036 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.608486 14429 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.608760 17112 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.609241 17126 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.610258 16780 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.610363 13926 tablet_copy_source_session.cc:205] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:30.610795 17112 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.613392 17154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:30.614753 13904 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.615272 14166 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:30.615682 16780 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.615932 17126 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:30.616683 17112 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.623023 14429 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.626547 14296 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.625483 13904 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.628042 17154 tablet_copy_client.cc:783] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:30.628314 17154 tablet_copy_client.cc:647] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:30.630072 14562 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.630684 13764 catalog_manager.cc:5484] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } }
I20240501 20:12:30.633996 14036 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.640904 17154 tablet_copy_client.cc:523] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:30.646420 14694 consensus_queue.cc:237] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:30.646811 17154 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:30.649330 14166 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:30.651722 13904 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.652256 14296 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.652889 17126 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.653239 14036 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.653343 16099 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.654071 16099 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.657008 14429 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.657590 17112 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.660367 17112 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:30.661062 14166 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.661798 16099 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.663110 14296 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:30.666543 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 77ef10335c1642818694b42ab006a66f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:30.667223 14429 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:30.672267 13759 catalog_manager.cc:5484] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:30.669775 14166 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:30.676297 14563 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:30.676227 13904 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:30.677299 16099 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:30.679078 14563 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:30.682065 14036 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
W20240501 20:12:30.688287 14630 consensus_peers.cc:485] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77ef10335c1642818694b42ab006a66f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:30.726117 17154 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:30.726612 17154 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:30.726936 17154 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.080s user 0.036s sys 0.008s
I20240501 20:12:30.727833 17154 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.728128 17154 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:30.728343 17154 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.730274 17154 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 20:12:30.731110 13926 tablet_copy_service.cc:342] P c12d6cddb7f242b6af650a10bbc8eb69: Request end of tablet copy session 1899220968384feebca3562bd7741a21-1840c628fc114b309da99d3c8a91656a received from {username='slave'} at 127.28.136.74:52915
I20240501 20:12:30.731292 13926 tablet_copy_service.cc:433] P c12d6cddb7f242b6af650a10bbc8eb69: ending tablet copy session 1899220968384feebca3562bd7741a21-1840c628fc114b309da99d3c8a91656a on tablet 1840c628fc114b309da99d3c8a91656a with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:30.755301 17154 ts_tablet_manager.cc:902] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:30.763468 17154 tablet_copy_client.cc:308] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:32991
I20240501 20:12:30.764348 14451 tablet_copy_service.cc:140] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Received BeginTabletCopySession request for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:58191)
I20240501 20:12:30.764521 14451 tablet_copy_service.cc:161] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Beginning new tablet copy session on tablet 9ae91bfd01694bea81cda5ffc9e7bc77 from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:58191: session id = 1899220968384feebca3562bd7741a21-9ae91bfd01694bea81cda5ffc9e7bc77
I20240501 20:12:30.767467 14451 tablet_copy_source_session.cc:205] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:30.771723 17154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:30.777770 17154 tablet_copy_client.cc:783] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:30.778038 17154 tablet_copy_client.cc:647] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:30.779644 17154 tablet_copy_client.cc:523] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:30.783113 17154 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:30.841105 17154 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:30.841612 17154 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:30.841877 17154 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.059s user 0.040s sys 0.004s
I20240501 20:12:30.842536 17154 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.842768 17154 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:30.842983 17154 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.843515 17154 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:30.844173 14451 tablet_copy_service.cc:342] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Request end of tablet copy session 1899220968384feebca3562bd7741a21-9ae91bfd01694bea81cda5ffc9e7bc77 received from {username='slave'} at 127.28.136.74:58191
I20240501 20:12:30.844336 14451 tablet_copy_service.cc:433] P 3b9f58142e4a4ccd9d0cbafc97d459c7: ending tablet copy session 1899220968384feebca3562bd7741a21-9ae91bfd01694bea81cda5ffc9e7bc77 on tablet 9ae91bfd01694bea81cda5ffc9e7bc77 with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:30.869155 17154 ts_tablet_manager.cc:902] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:30.874742 17154 tablet_copy_client.cc:308] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:30.881642 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet 56c262e138874431a07f6d0bfcb516bf from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:35471)
I20240501 20:12:30.881817 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet 56c262e138874431a07f6d0bfcb516bf from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:35471: session id = 1899220968384feebca3562bd7741a21-56c262e138874431a07f6d0bfcb516bf
I20240501 20:12:30.883852 14714 tablet_copy_source_session.cc:205] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:30.885092 17154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:30.889818 17154 tablet_copy_client.cc:783] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:30.890055 17154 tablet_copy_client.cc:647] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:30.891584 17154 tablet_copy_client.cc:523] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:30.894353 17154 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:30.924917 17154 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:30.925411 17154 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:30.925724 17154 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.031s user 0.032s sys 0.000s
I20240501 20:12:30.926472 17154 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.926754 17154 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:30.926985 17154 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:30.927580 17154 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:30.928472 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session 1899220968384feebca3562bd7741a21-56c262e138874431a07f6d0bfcb516bf received from {username='slave'} at 127.28.136.74:35471
I20240501 20:12:30.928632 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session 1899220968384feebca3562bd7741a21-56c262e138874431a07f6d0bfcb516bf on tablet 56c262e138874431a07f6d0bfcb516bf with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:30.987931 14036 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:30.990383 14694 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.990422 14563 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.990486 14296 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.990473 16304 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.991103 15769 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.991612 14166 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.992069 16619 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.992367 16811 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.992550 13904 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:30.993237 16944 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.993592 17230 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.993956 16811 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:30.997869 16619 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.998363 14563 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:30.998756 14166 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:31.001739 13904 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:31.002087 14296 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:31.005887 13762 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:31.006824 13749 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe1592bebd6b4a858987847446f15a16 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:31.007804 14694 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:31.013520 14410 tablet_service.cc:1514] Processing DeleteTablet for tablet fe1592bebd6b4a858987847446f15a16 with delete_type TABLET_DATA_TOMBSTONED (TS 3b9f58142e4a4ccd9d0cbafc97d459c7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54284
I20240501 20:12:31.017549 16304 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:31.021682 17234 tablet_replica.cc:330] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: stopping tablet replica
I20240501 20:12:31.024255 17234 raft_consensus.cc:2229] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:31.024699 17234 raft_consensus.cc:2258] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:31.025790 17234 ts_tablet_manager.cc:1875] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:31.032004 17234 ts_tablet_manager.cc:1888] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:31.032151 17234 log.cc:1196] T fe1592bebd6b4a858987847446f15a16 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/wals/fe1592bebd6b4a858987847446f15a16
I20240501 20:12:31.032955 13751 catalog_manager.cc:4830] TS 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): tablet fe1592bebd6b4a858987847446f15a16 (table rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]) successfully deleted
I20240501 20:12:31.033650 14036 consensus_queue.cc:237] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:31.036465 14296 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:31.036476 14429 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:31.036487 14694 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:31.037045 16944 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:31.037547 16619 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:31.037612 14563 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:31.037830 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:31.040120 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:31.041903 14166 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:31.041677 16619 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:31.042510 16944 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:31.044327 14166 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:31.045316 13904 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:31.046641 14298 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:31.048084 16944 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:31.050316 14563 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:31.050882 13762 catalog_manager.cc:5484] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:31.052445 14694 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:31.054136 14429 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:31.059688 13904 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:31.064558 14036 consensus_queue.cc:237] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:31.068027 14298 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:12:31.068763 13974 consensus_peers.cc:485] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e126e98f1349cdb3e532ff49d3e926. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:31.068835 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:31.070461 14694 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:31.071132 16619 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:31.071223 14166 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:31.071586 14562 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:31.072134 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:31.073127 13905 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:31.074216 16436 raft_consensus.cc:1203] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:31.078554 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:31.081496 16811 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:31.085608 16619 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:31.087153 14562 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:31.091059 13749 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 58e126e98f1349cdb3e532ff49d3e926 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:31.092717 13762 catalog_manager.cc:5484] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:31.093839 14298 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:31.096212 14429 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:31.097107 16811 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:31.100931 14429 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:31.101475 14694 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:31.103971 13904 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:31.108350 14166 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:31.117229 17154 ts_tablet_manager.cc:902] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:31.132339 17154 tablet_copy_client.cc:308] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38579
I20240501 20:12:31.139786 14189 tablet_copy_service.cc:140] P 0e898d8ee23441908d1593c561b368ad: Received BeginTabletCopySession request for tablet 3808b45349de4959b929a99735e5db53 from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:58907)
I20240501 20:12:31.140079 14189 tablet_copy_service.cc:161] P 0e898d8ee23441908d1593c561b368ad: Beginning new tablet copy session on tablet 3808b45349de4959b929a99735e5db53 from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:58907: session id = 1899220968384feebca3562bd7741a21-3808b45349de4959b929a99735e5db53
I20240501 20:12:31.143899 14189 tablet_copy_source_session.cc:205] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:31.145836 17154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:31.153595 17154 tablet_copy_client.cc:783] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:31.153810 17154 tablet_copy_client.cc:647] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:31.155895 17154 tablet_copy_client.cc:523] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:31.159018 17154 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:31.198010 17154 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:31.198457 17154 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:31.198758 17154 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.040s user 0.029s sys 0.009s
I20240501 20:12:31.199577 17154 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:31.199940 17154 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:31.200170 17154 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:31.200794 17154 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 20:12:31.201745 14189 tablet_copy_service.cc:342] P 0e898d8ee23441908d1593c561b368ad: Request end of tablet copy session 1899220968384feebca3562bd7741a21-3808b45349de4959b929a99735e5db53 received from {username='slave'} at 127.28.136.74:58907
I20240501 20:12:31.201921 14189 tablet_copy_service.cc:433] P 0e898d8ee23441908d1593c561b368ad: ending tablet copy session 1899220968384feebca3562bd7741a21-3808b45349de4959b929a99735e5db53 on tablet 3808b45349de4959b929a99735e5db53 with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:31.234293 17288 ts_tablet_manager.cc:902] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:31.245170 17288 tablet_copy_client.cc:308] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:31.252938 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet 77ef10335c1642818694b42ab006a66f from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:33909)
I20240501 20:12:31.253126 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet 77ef10335c1642818694b42ab006a66f from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:33909: session id = aeb70769b705493bb5196527311b142b-77ef10335c1642818694b42ab006a66f
I20240501 20:12:31.255322 14714 tablet_copy_source_session.cc:205] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:31.256762 17288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:31.261401 17288 tablet_copy_client.cc:783] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:31.261737 17288 tablet_copy_client.cc:647] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:31.263432 17288 tablet_copy_client.cc:523] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:31.265928 17288 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:31.271431 16436 raft_consensus.cc:1203] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:31.281018 16436 raft_consensus.cc:1203] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:31.307569 17288 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:31.308005 17288 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:31.308331 17288 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.042s user 0.032s sys 0.001s
I20240501 20:12:31.309193 17288 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:31.309459 17288 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:31.309706 17288 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:31.310338 17288 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:12:31.311520 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session aeb70769b705493bb5196527311b142b-77ef10335c1642818694b42ab006a66f received from {username='slave'} at 127.28.136.72:33909
I20240501 20:12:31.311650 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session aeb70769b705493bb5196527311b142b-77ef10335c1642818694b42ab006a66f on tablet 77ef10335c1642818694b42ab006a66f with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:31.543249 16436 raft_consensus.cc:1203] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:31.631086 17081 raft_consensus.cc:1050] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:31.632369 17081 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:31.636138 14166 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.636265 16436 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.636842 14692 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.636849 17326 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.637432 17328 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.638042 17328 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.646222 14036 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.646226 14429 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.646801 14562 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.646872 17328 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.647771 14298 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.648970 17328 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.649622 17328 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.650099 17328 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.652683 17335 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.653388 14036 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.655019 14562 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.656123 14298 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.662809 13758 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.672212 14429 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.679065 14692 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.679065 14166 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.679329 16436 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.685426 17288 ts_tablet_manager.cc:902] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:31.693729 17288 tablet_copy_client.cc:308] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:31.694515 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet 58e126e98f1349cdb3e532ff49d3e926 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:40007)
I20240501 20:12:31.694679 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet 58e126e98f1349cdb3e532ff49d3e926 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:40007: session id = aeb70769b705493bb5196527311b142b-58e126e98f1349cdb3e532ff49d3e926
I20240501 20:12:31.697499 14058 tablet_copy_source_session.cc:205] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:31.699648 17288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e126e98f1349cdb3e532ff49d3e926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:31.701875 13904 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:31.705053 17288 tablet_copy_client.cc:783] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:31.705286 17288 tablet_copy_client.cc:647] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:31.706902 14036 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.707605 14166 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.707939 14298 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.707957 16435 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.708709 14429 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.708823 17112 raft_consensus.cc:1050] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:31.708776 17335 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.710039 17112 consensus_queue.cc:237] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:31.710427 17344 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.710922 17344 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.711460 17344 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.711529 14692 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.711866 17335 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.715123 17081 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.716149 17344 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.717036 14036 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.718319 14692 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.720011 17288 tablet_copy_client.cc:523] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:31.720429 13752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1840c628fc114b309da99d3c8a91656a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:31.722113 16435 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.722128 14298 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.722232 14166 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.722807 13761 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.733563 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 1840c628fc114b309da99d3c8a91656a with delete_type TABLET_DATA_TOMBSTONED (TS a5ca96c8a9a74413bff544306af10586 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59144
I20240501 20:12:31.740195 17288 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:31.749436 17352 tablet_replica.cc:330] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: stopping tablet replica
I20240501 20:12:31.749436 16304 raft_consensus.cc:1203] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:31.749927 17352 raft_consensus.cc:2229] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:31.750330 17352 raft_consensus.cc:2258] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:31.751413 17352 ts_tablet_manager.cc:1875] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:31.763064 16435 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.764312 14562 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.765102 14429 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.768253 17356 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.770455 17356 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.772912 17352 ts_tablet_manager.cc:1888] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:31.775408 17352 log.cc:1196] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/wals/1840c628fc114b309da99d3c8a91656a
I20240501 20:12:31.776979 13750 catalog_manager.cc:4830] TS a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): tablet 1840c628fc114b309da99d3c8a91656a (table rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]) successfully deleted
I20240501 20:12:31.807969 14298 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.809085 17354 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:12:31.827018 14627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:44849, user_credentials={real_user=slave}} blocked reactor thread for 61320.6us
W20240501 20:12:31.827476 14630 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.73:39773, user_credentials={real_user=slave}} blocked reactor thread for 62151.2us
I20240501 20:12:31.844578 14036 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.844578 14429 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.845229 14166 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.845260 17126 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.845679 17356 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.846192 17356 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.854486 13904 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.856928 17126 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.861187 17126 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.865850 14296 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.865909 13904 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.872447 13757 catalog_manager.cc:5484] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.875329 14562 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.876498 14036 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.882112 16436 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.884467 14166 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.889562 14692 consensus_queue.cc:237] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:31.895514 14169 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.896030 16436 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.896971 17112 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.897506 17112 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.900118 14036 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.900118 13904 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.900827 17112 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.901124 17126 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.905268 17355 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.909271 14562 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.909334 16435 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.909821 17112 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.911321 14563 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.915237 17286 raft_consensus.cc:1050] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:31.917187 13757 catalog_manager.cc:5484] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 reported cstate change: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.919629 13904 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.921053 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56c262e138874431a07f6d0bfcb516bf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:31.922749 14035 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.923305 14410 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c262e138874431a07f6d0bfcb516bf with delete_type TABLET_DATA_TOMBSTONED (TS 3b9f58142e4a4ccd9d0cbafc97d459c7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54284
I20240501 20:12:31.925015 17286 consensus_queue.cc:237] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:31.928661 14298 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:31.937901 17355 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:31.941463 14169 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.940127 14298 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:31.967901 17370 tablet_replica.cc:330] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: stopping tablet replica
I20240501 20:12:31.968237 17370 raft_consensus.cc:2229] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:31.968603 17370 pending_rounds.cc:62] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: Trying to abort 1 pending ops.
I20240501 20:12:31.968782 17370 pending_rounds.cc:68] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022978464600903680 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "56c262e138874431a07f6d0bfcb516bf" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } } }
I20240501 20:12:31.969497 17370 raft_consensus.cc:2894] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 20:12:31.969681 17370 raft_consensus.cc:2866] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:12:31.969835 17370 raft_consensus.cc:2258] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:31.970860 17370 ts_tablet_manager.cc:1875] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:31.971434 16435 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.972038 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.974836 14169 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.975519 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.977607 14035 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.977682 13904 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.978226 14692 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:31.978253 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.978724 17373 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.979815 17373 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:31.997370 17370 ts_tablet_manager.cc:1888] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:31.998420 17370 log.cc:1196] T 56c262e138874431a07f6d0bfcb516bf P 3b9f58142e4a4ccd9d0cbafc97d459c7: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-4/wal/wals/56c262e138874431a07f6d0bfcb516bf
I20240501 20:12:31.999531 13751 catalog_manager.cc:4830] TS 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991): tablet 56c262e138874431a07f6d0bfcb516bf (table rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]) successfully deleted
I20240501 20:12:32.004802 14562 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.005457 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.012238 14298 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.016292 17286 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.031931 17373 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.036252 17288 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:32.037701 17288 tablet_bootstrap.cc:492] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:32.038702 17288 ts_tablet_manager.cc:1367] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.299s user 0.044s sys 0.008s
I20240501 20:12:32.040069 17288 raft_consensus.cc:348] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:32.040407 17288 raft_consensus.cc:729] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:32.041419 14296 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.041941 17288 consensus_queue.cc:260] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:32.043979 17288 ts_tablet_manager.cc:1398] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:12:32.044692 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session aeb70769b705493bb5196527311b142b-58e126e98f1349cdb3e532ff49d3e926 received from {username='slave'} at 127.28.136.72:40007
I20240501 20:12:32.044863 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session aeb70769b705493bb5196527311b142b-58e126e98f1349cdb3e532ff49d3e926 on tablet 58e126e98f1349cdb3e532ff49d3e926 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:32.048478 16435 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.053222 13757 catalog_manager.cc:5484] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.055529 14563 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.068892 14692 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.084372 13905 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.087803 14169 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.092087 14035 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.092820 14429 consensus_queue.cc:237] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:32.095785 16435 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.095950 14169 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.096441 14562 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.096642 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.097056 17371 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.097810 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.099114 13904 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.100072 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.100464 14692 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.101109 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.104171 14036 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.105119 17369 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.106972 17373 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.107661 14562 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.112013 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:32.111649 14036 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.114869 14692 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.115800 13763 catalog_manager.cc:5484] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:32.119683 16435 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.121760 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:32.121618 13904 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.129976 14166 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.139817 17396 tablet_replica.cc:330] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:32.140213 17396 raft_consensus.cc:2229] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:32.140794 17396 raft_consensus.cc:2258] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:32.142010 17396 ts_tablet_manager.cc:1875] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:32.148047 17396 ts_tablet_manager.cc:1888] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:32.148208 17396 log.cc:1196] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/9ae91bfd01694bea81cda5ffc9e7bc77
I20240501 20:12:32.149003 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (table rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]) successfully deleted
I20240501 20:12:32.157022 16166 raft_consensus.cc:1050] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:32.158119 16166 consensus_queue.cc:237] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:32.161628 13905 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.161932 16435 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.162325 14035 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.163239 17355 raft_consensus.cc:1050] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:32.163658 14692 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.163724 14429 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.164456 17355 consensus_queue.cc:237] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:32.162508 17410 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240501 20:12:32.169267 17390 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.170183 17390 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.170745 17390 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.179481 14563 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.179944 17412 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.180114 17179 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.199576 16304 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.201649 17179 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.206064 14562 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.206104 14298 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.206732 17412 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.207010 14298 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.207304 17411 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.208868 17179 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.210702 17390 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.211833 14429 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.212507 17112 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.213618 13904 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.214429 17355 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.214625 14036 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.215425 17112 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.215502 14169 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.217250 17126 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.242911 16435 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.247202 14692 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.253949 13764 catalog_manager.cc:5484] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 reported cstate change: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.263126 14298 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.273845 17126 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.277534 16304 raft_consensus.cc:1203] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:32.283003 14430 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.282975 13904 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.283572 13763 catalog_manager.cc:5484] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:32.286672 16303 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.288404 14035 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.290216 14166 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.288393 14429 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.307412 14562 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.309111 14563 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.282150 14296 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.317914 13904 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.319399 14166 consensus_queue.cc:237] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:32.322767 14298 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.322901 16435 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.323191 14429 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.323410 17411 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.323549 14692 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.324095 17410 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.324386 14035 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.324594 17410 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.325171 17432 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.326866 17390 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.329396 13905 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.334436 16435 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.341832 13762 catalog_manager.cc:5484] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 reported cstate change: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.333253 17413 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.347502 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3808b45349de4959b929a99735e5db53 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:32.350086 14296 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.355340 14429 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.360653 14037 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.364933 17429 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.365175 14035 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.368459 14692 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.373893 14694 consensus_queue.cc:237] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:32.377741 16304 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.379696 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet 3808b45349de4959b929a99735e5db53 with delete_type TABLET_DATA_TOMBSTONED (TS a5ca96c8a9a74413bff544306af10586 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59144
I20240501 20:12:32.381091 14563 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.381507 13905 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.384064 17126 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.384671 17126 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.385367 17355 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.393236 17179 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.399168 14429 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.399317 16304 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.399880 17126 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.402172 14429 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.405548 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 77ef10335c1642818694b42ab006a66f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:32.405366 13763 catalog_manager.cc:5484] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.409341 14563 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.410102 13905 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.413391 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ef10335c1642818694b42ab006a66f with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:32.414572 17396 tablet_replica.cc:330] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:32.414801 17396 raft_consensus.cc:2229] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:32.415102 17396 raft_consensus.cc:2258] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:32.416399 17396 ts_tablet_manager.cc:1875] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:12:32.420414 14103 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.68:38031, user_credentials={real_user=slave}} blocked reactor thread for 49428.5us
I20240501 20:12:32.421213 13905 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:32.422441 14166 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.423789 17112 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.425899 17396 ts_tablet_manager.cc:1888] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:32.426026 17396 log.cc:1196] T 77ef10335c1642818694b42ab006a66f P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/77ef10335c1642818694b42ab006a66f
I20240501 20:12:32.426049 14169 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.426657 17438 tablet_replica.cc:330] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: stopping tablet replica
I20240501 20:12:32.427013 17438 raft_consensus.cc:2229] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:32.427321 17438 raft_consensus.cc:2258] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:32.428565 14036 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.429380 17179 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.431306 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet 77ef10335c1642818694b42ab006a66f (table rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]) successfully deleted
I20240501 20:12:32.431214 14036 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.441053 17438 ts_tablet_manager.cc:1875] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:32.446007 17438 ts_tablet_manager.cc:1888] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:32.446137 17438 log.cc:1196] T 3808b45349de4959b929a99735e5db53 P a5ca96c8a9a74413bff544306af10586: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-5/wal/wals/3808b45349de4959b929a99735e5db53
I20240501 20:12:32.447295 13750 catalog_manager.cc:4830] TS a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): tablet 3808b45349de4959b929a99735e5db53 (table rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]) successfully deleted
I20240501 20:12:32.530292 14169 consensus_queue.cc:237] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:32.533304 14037 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.533491 13905 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.533608 14563 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.534015 17429 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.534226 14296 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.534325 17390 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.534932 17390 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.535602 14694 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.536196 14429 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.537793 17413 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.538305 17429 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.538594 17411 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.542064 17429 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.542867 14562 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.543759 14694 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.545364 14429 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.548432 14037 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.550668 13905 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.554033 13764 catalog_manager.cc:5484] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.564680 14169 consensus_queue.cc:237] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:32.568133 14037 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.568750 14298 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.568989 17411 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:32.569466 17432 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.570281 13905 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.570688 14562 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.570748 14296 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.571419 14694 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.572024 14429 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.572944 17429 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:32.573204 17432 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:32.573652 17413 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:32.577416 17432 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:32.578485 14298 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:32.581640 13904 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:32.584648 13762 catalog_manager.cc:5484] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
W20240501 20:12:32.586652 14105 consensus_peers.cc:485] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d09003d74074293b86f2390f25ee58a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:32.586467 14563 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:32.587010 14429 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:32.577020 17390 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:32.590593 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7d09003d74074293b86f2390f25ee58a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:32.592931 14037 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:32.600219 14694 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:32.771335 16619 raft_consensus.cc:1050] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:32.772272 16619 consensus_queue.cc:237] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:32.775960 14694 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.776472 14563 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.776768 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.777455 14429 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.777745 17475 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.779145 14298 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.779778 17476 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.779930 16304 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.781049 13904 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.781150 17476 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.781664 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.783381 14169 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:32.785683 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.786846 16619 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:32.787333 17475 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.788378 16304 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.794114 14694 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.796926 13764 catalog_manager.cc:5484] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b reported cstate change: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.798031 14563 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.806584 14296 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.831948 14169 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.832454 13904 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.845309 14037 consensus_queue.cc:237] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:32.849314 16304 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.850989 14563 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.851584 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.852315 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.854318 14694 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.854662 14166 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.856139 13904 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.856608 17476 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.856951 16619 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.859637 17476 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.862375 16619 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.852169 14429 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.863973 14694 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.865795 13905 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.867415 13749 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 58e126e98f1349cdb3e532ff49d3e926 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:32.870615 14563 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.873122 14168 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.875748 16304 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.886322 13763 catalog_manager.cc:5484] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.894095 14430 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:32.896392 17230 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:32.899936 14430 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:32.904151 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e126e98f1349cdb3e532ff49d3e926 with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:32.904565 17396 tablet_replica.cc:330] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:32.906018 17396 raft_consensus.cc:2229] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:32.906276 17396 raft_consensus.cc:2258] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:32.907985 17396 ts_tablet_manager.cc:1875] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:32.915303 17396 ts_tablet_manager.cc:1888] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:32.915446 17396 log.cc:1196] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/58e126e98f1349cdb3e532ff49d3e926
I20240501 20:12:32.916236 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet 58e126e98f1349cdb3e532ff49d3e926 (table rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]) successfully deleted
I20240501 20:12:32.928153 14694 consensus_queue.cc:237] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:32.931810 14296 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.932507 13905 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.933362 17126 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.933913 14563 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.934340 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.936703 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.938370 14168 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.941793 17126 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.945811 14037 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.946552 17492 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.946694 14430 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:32.947435 17126 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:32.961266 17126 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.964824 14168 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.964989 14430 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.966459 14037 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.970350 13904 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.971228 13763 catalog_manager.cc:5484] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.975682 14296 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:32.980012 14694 consensus_queue.cc:237] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:32.986822 14169 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.987859 17126 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:32.989315 14037 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.991487 17112 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:12:32.996971 14630 consensus_peers.cc:485] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5fa6ed23f1f44b5b9fa96d54ed22ee6. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:32.997747 14296 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:32.998310 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.001703 17492 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:33.005620 14298 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:33.012226 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:33.012045 14169 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:33.013052 14037 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:33.018910 13762 catalog_manager.cc:5484] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:33.021291 14430 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:33.021939 17126 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.023579 14429 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:33.029809 13905 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:33.029888 14562 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:33.032287 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.032971 17126 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:33.034324 14562 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.036845 13904 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:33.039142 14562 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:33.192644 17514 ts_tablet_manager.cc:902] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:33.194377 17514 tablet_copy_client.cc:308] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38579
I20240501 20:12:33.200433 14189 tablet_copy_service.cc:140] P 0e898d8ee23441908d1593c561b368ad: Received BeginTabletCopySession request for tablet 7d09003d74074293b86f2390f25ee58a from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:35535)
I20240501 20:12:33.200641 14189 tablet_copy_service.cc:161] P 0e898d8ee23441908d1593c561b368ad: Beginning new tablet copy session on tablet 7d09003d74074293b86f2390f25ee58a from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:35535: session id = aeb70769b705493bb5196527311b142b-7d09003d74074293b86f2390f25ee58a
I20240501 20:12:33.202903 14189 tablet_copy_source_session.cc:205] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:33.204233 17514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d09003d74074293b86f2390f25ee58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:33.208988 17514 tablet_copy_client.cc:783] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:33.209233 17514 tablet_copy_client.cc:647] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:33.211052 17514 tablet_copy_client.cc:523] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:33.213943 17514 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:33.252342 17514 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:33.253243 17514 tablet_bootstrap.cc:492] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:33.253935 17514 ts_tablet_manager.cc:1367] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.040s user 0.034s sys 0.005s
I20240501 20:12:33.254838 17514 raft_consensus.cc:348] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:33.255136 17514 raft_consensus.cc:729] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:33.255360 17514 consensus_queue.cc:260] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:33.255954 17514 ts_tablet_manager.cc:1398] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:33.257174 14189 tablet_copy_service.cc:342] P 0e898d8ee23441908d1593c561b368ad: Request end of tablet copy session aeb70769b705493bb5196527311b142b-7d09003d74074293b86f2390f25ee58a received from {username='slave'} at 127.28.136.72:35535
I20240501 20:12:33.257318 14189 tablet_copy_service.cc:433] P 0e898d8ee23441908d1593c561b368ad: ending tablet copy session aeb70769b705493bb5196527311b142b-7d09003d74074293b86f2390f25ee58a on tablet 7d09003d74074293b86f2390f25ee58a with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:33.290118 14169 consensus_queue.cc:237] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:33.292702 14296 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:33.292678 14694 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:33.292726 14562 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:33.293393 17413 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:33.293993 14037 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:33.294046 13904 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:33.294683 17390 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:33.294999 17429 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:33.297045 17390 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:33.297737 17429 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:33.311133 17429 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.315316 14037 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.318118 14298 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.323621 13762 catalog_manager.cc:5484] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.329164 14430 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:33.330816 17527 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:33.330780 14562 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.332949 13904 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.333405 14430 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.339409 14169 consensus_queue.cc:237] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:33.346392 14430 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:33.346789 14037 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:33.347329 14296 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 20:12:33.347014 14104 consensus_peers.cc:485] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39eb086c2b80482a98ff2baae96b9f60. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:33.347793 17527 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.349269 17527 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.349509 13904 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:33.350281 17413 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.350455 14563 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:33.352167 17390 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.353945 17527 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.356524 17413 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:33.362840 14430 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:33.364850 13751 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 39eb086c2b80482a98ff2baae96b9f60 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:33.363886 13904 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:33.365833 14296 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:33.367201 14037 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:33.367336 14562 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:33.370925 13762 catalog_manager.cc:5484] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:33.375615 14694 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:33.379839 14692 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:33.391173 17390 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:33.392966 14694 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:33.502862 17514 ts_tablet_manager.cc:902] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:33.504073 17514 tablet_copy_client.cc:308] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:33.504813 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:33909)
I20240501 20:12:33.504971 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:33909: session id = aeb70769b705493bb5196527311b142b-c5fa6ed23f1f44b5b9fa96d54ed22ee6
I20240501 20:12:33.506982 14714 tablet_copy_source_session.cc:205] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:33.508407 17514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:33.513258 17514 tablet_copy_client.cc:783] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:33.513499 17514 tablet_copy_client.cc:647] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:33.515457 17514 tablet_copy_client.cc:523] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:33.518388 17514 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:33.550127 17514 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:33.550697 17514 tablet_bootstrap.cc:492] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:33.551005 17514 ts_tablet_manager.cc:1367] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20240501 20:12:33.551703 17514 raft_consensus.cc:348] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:33.551939 17514 raft_consensus.cc:729] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:33.552145 17514 consensus_queue.cc:260] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:33.552692 17514 ts_tablet_manager.cc:1398] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:33.553475 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session aeb70769b705493bb5196527311b142b-c5fa6ed23f1f44b5b9fa96d54ed22ee6 received from {username='slave'} at 127.28.136.72:33909
I20240501 20:12:33.553606 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session aeb70769b705493bb5196527311b142b-c5fa6ed23f1f44b5b9fa96d54ed22ee6 on tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:33.628374 16304 raft_consensus.cc:1203] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:33.792533 17572 ts_tablet_manager.cc:902] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:33.794857 17572 tablet_copy_client.cc:308] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38579
I20240501 20:12:33.795738 14189 tablet_copy_service.cc:140] P 0e898d8ee23441908d1593c561b368ad: Received BeginTabletCopySession request for tablet 39eb086c2b80482a98ff2baae96b9f60 from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:58907)
I20240501 20:12:33.795917 14189 tablet_copy_service.cc:161] P 0e898d8ee23441908d1593c561b368ad: Beginning new tablet copy session on tablet 39eb086c2b80482a98ff2baae96b9f60 from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:58907: session id = 1899220968384feebca3562bd7741a21-39eb086c2b80482a98ff2baae96b9f60
I20240501 20:12:33.798209 14189 tablet_copy_source_session.cc:205] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:33.799531 17572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:33.805477 17572 tablet_copy_client.cc:783] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:33.805750 17572 tablet_copy_client.cc:647] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:33.807260 17572 tablet_copy_client.cc:523] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:33.809933 17572 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:33.849399 17572 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:33.849864 17572 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:33.850158 17572 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.040s user 0.035s sys 0.004s
I20240501 20:12:33.850955 17572 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:33.851280 17572 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:33.851501 17572 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:33.852473 17572 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:12:33.853355 14189 tablet_copy_service.cc:342] P 0e898d8ee23441908d1593c561b368ad: Request end of tablet copy session 1899220968384feebca3562bd7741a21-39eb086c2b80482a98ff2baae96b9f60 received from {username='slave'} at 127.28.136.74:58907
I20240501 20:12:33.853502 14189 tablet_copy_service.cc:433] P 0e898d8ee23441908d1593c561b368ad: ending tablet copy session 1899220968384feebca3562bd7741a21-39eb086c2b80482a98ff2baae96b9f60 on tablet 39eb086c2b80482a98ff2baae96b9f60 with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:33.893553 16304 raft_consensus.cc:1203] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:34.061641 17532 raft_consensus.cc:1050] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:34.062841 17532 consensus_queue.cc:237] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:34.066118 14692 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.066095 14562 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.066702 14296 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.066772 17536 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.067410 17536 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.067926 17390 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.068447 16304 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.069497 14430 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.070206 17536 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.072824 17536 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.077340 14037 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.078042 17596 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.078495 13905 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.079475 17390 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.094702 17596 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.095376 13905 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.097232 14296 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.099494 14692 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.099848 16304 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.101415 14562 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.104260 14430 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.106940 13757 catalog_manager.cc:5484] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:34.108254 14036 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.132477 14168 consensus_queue.cc:237] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:34.136782 14037 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.137338 16304 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.137461 14692 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.137552 17532 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.138373 13905 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.138833 14430 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.139230 17536 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.139796 17595 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.140381 17390 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.141021 17390 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.143811 17536 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.145447 14430 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.145960 13905 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.148077 14037 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.150169 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d09003d74074293b86f2390f25ee58a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:34.152390 13758 catalog_manager.cc:5484] T 7d09003d74074293b86f2390f25ee58a P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.153688 16304 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.156154 14562 raft_consensus.cc:1261] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.156751 17390 consensus_queue.cc:1035] T 7d09003d74074293b86f2390f25ee58a P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.158119 14692 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.159655 14562 raft_consensus.cc:2932] T 7d09003d74074293b86f2390f25ee58a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.166993 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d09003d74074293b86f2390f25ee58a with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:34.167752 17619 tablet_replica.cc:330] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:34.167974 17619 raft_consensus.cc:2229] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:34.168327 17619 raft_consensus.cc:2258] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:34.169181 17619 ts_tablet_manager.cc:1875] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:34.174957 17619 ts_tablet_manager.cc:1888] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:34.175114 17619 log.cc:1196] T 7d09003d74074293b86f2390f25ee58a P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d09003d74074293b86f2390f25ee58a
I20240501 20:12:34.177098 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet 7d09003d74074293b86f2390f25ee58a (table rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]) successfully deleted
I20240501 20:12:34.265527 16435 raft_consensus.cc:1203] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:34.293138 17126 raft_consensus.cc:1050] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:34.294445 17126 consensus_queue.cc:237] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:34.297894 16304 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.298074 14562 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.298640 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.298789 14296 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.299625 17556 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.300220 17556 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.303998 14430 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.305301 13905 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.305946 14168 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.306892 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.307487 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.307947 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.313272 14037 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.313966 17556 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.324771 17556 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.328054 14037 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.328553 14296 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.329342 14168 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.329879 14562 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.334296 13760 catalog_manager.cc:5484] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.338297 13905 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.340833 16304 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.346596 14430 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.352715 14692 consensus_queue.cc:237] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:34.358794 14562 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.359304 13905 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.359799 16304 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.360386 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.361019 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.361326 14037 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.362150 14168 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.362805 17634 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.363123 14429 raft_consensus.cc:1261] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.364300 17126 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.364791 17636 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.366150 17179 consensus_queue.cc:1035] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.367873 17556 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.370894 14430 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.371001 14168 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.373188 14037 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.376497 13763 catalog_manager.cc:5484] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.380120 13905 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.380168 14562 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.382344 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:34.386184 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:34.385735 16304 raft_consensus.cc:2932] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:34.386579 17619 tablet_replica.cc:330] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:34.386780 17619 raft_consensus.cc:2229] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:34.387187 17619 raft_consensus.cc:2258] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:34.388916 17619 ts_tablet_manager.cc:1875] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:34.401803 17619 ts_tablet_manager.cc:1888] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:34.402076 17619 log.cc:1196] T c5fa6ed23f1f44b5b9fa96d54ed22ee6 P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5fa6ed23f1f44b5b9fa96d54ed22ee6
I20240501 20:12:34.403077 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet c5fa6ed23f1f44b5b9fa96d54ed22ee6 (table rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]) successfully deleted
I20240501 20:12:34.417057 14562 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:34.420130 14037 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.420346 13905 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.421123 14296 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.421224 14692 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.421293 14169 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.421391 14430 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.421757 17437 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.422207 17613 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.423655 17613 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.424115 17613 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.424561 17613 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.424954 17437 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.455900 17613 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.464505 14694 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.466396 14296 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.466639 13761 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:34.469504 13905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.481122 14169 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.481493 14037 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.484347 14562 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
W20240501 20:12:34.491700 14498 consensus_peers.cc:485] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eeaff69149ee4707beba512799b0d0df. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:34.492092 14296 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.492659 13905 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.492710 14694 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.493405 17647 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.493762 14169 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.494161 14037 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.494818 17437 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.495278 17437 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.495854 17645 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.496343 17613 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.500178 14430 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.500994 17647 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.501928 14037 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.503872 17613 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.503898 14169 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.505417 14694 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.512449 13750 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet eeaff69149ee4707beba512799b0d0df with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:34.513865 14296 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.514367 13762 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.510120 14430 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.515596 13905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.523337 14430 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.763438 14562 consensus_queue.cc:237] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } }
I20240501 20:12:34.766510 14169 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.767170 14037 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.767215 13905 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.767735 17645 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.767890 14430 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.768543 17645 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.769627 17437 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.770308 17647 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.771149 14296 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.771906 17613 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.774698 14694 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:34.779361 17437 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.783429 17647 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.785398 17390 raft_consensus.cc:1050] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:34.786607 17390 consensus_queue.cc:237] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:34.787541 14296 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.790145 16435 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.791023 17536 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.791772 14692 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.791347 14694 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.793471 17536 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.799137 14298 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.800019 14036 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.800030 17536 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.800171 14563 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.800868 13905 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.801082 17596 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.801607 17596 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.801748 14429 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:34.803375 17536 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.808640 17536 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:34.816579 13761 catalog_manager.cc:5484] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.817054 13904 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.822705 14169 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.830546 14560 consensus_queue.cc:237] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:34.841487 14296 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.842187 17692 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.842509 14430 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.843153 17613 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:12:34.843792 14498 consensus_peers.cc:485] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59fdeaff3d92460c8ca59eee1144b127. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:34.845201 13904 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.845721 14431 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.849615 17613 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.855316 14169 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.859406 17647 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.862506 17613 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.865556 14431 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.868903 13904 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.869520 13750 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 59fdeaff3d92460c8ca59eee1144b127 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:34.872501 14035 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.874795 14295 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.883216 13760 catalog_manager.cc:5484] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.888073 17692 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:34.889700 14037 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } }
I20240501 20:12:34.895129 17536 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.902004 14037 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.906873 14035 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.909400 14692 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.921428 13762 catalog_manager.cc:5484] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.923331 13905 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.926177 14298 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.946830 14694 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:34.947741 14169 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.952709 17647 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:34.959300 14694 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:34.975442 14429 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.978928 16435 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.986621 14169 consensus_queue.cc:237] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:34.986361 14563 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:34.991180 14035 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.992375 13905 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.993016 14431 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.993681 17596 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.994211 17536 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.994500 17390 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:34.996472 14562 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:34.997311 17390 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:35.000401 14694 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:35.001410 17390 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:35.007179 17390 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:35.008704 16434 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:35.013163 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39eb086c2b80482a98ff2baae96b9f60 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:35.013121 17596 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:35.013701 14035 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:35.015965 14431 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:35.018221 14694 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:35.020615 14563 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:35.023175 16435 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:35.024245 13905 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:35.025007 13760 catalog_manager.cc:5484] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:35.029871 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 39eb086c2b80482a98ff2baae96b9f60 with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:35.033972 17699 tablet_replica.cc:330] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:35.034474 17699 raft_consensus.cc:2229] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:35.034811 17699 raft_consensus.cc:2258] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:35.037227 17699 ts_tablet_manager.cc:1875] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:35.045935 17699 ts_tablet_manager.cc:1888] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:35.046097 17699 log.cc:1196] T 39eb086c2b80482a98ff2baae96b9f60 P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/39eb086c2b80482a98ff2baae96b9f60
I20240501 20:12:35.046919 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet 39eb086c2b80482a98ff2baae96b9f60 (table rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]) successfully deleted
I20240501 20:12:35.116444 17711 ts_tablet_manager.cc:902] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:35.117789 17711 tablet_copy_client.cc:308] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:35.118532 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet eeaff69149ee4707beba512799b0d0df from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:37299)
I20240501 20:12:35.118738 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet eeaff69149ee4707beba512799b0d0df from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:37299: session id = 2a31af84f57c4ef18f0d98ff044229b6-eeaff69149ee4707beba512799b0d0df
I20240501 20:12:35.121941 14583 tablet_copy_source_session.cc:205] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:35.124150 17711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:35.131814 17711 tablet_copy_client.cc:783] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:35.132380 17711 tablet_copy_client.cc:647] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:35.134194 17711 tablet_copy_client.cc:523] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:35.136996 17711 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:35.184163 17711 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:35.184670 17711 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:35.185005 17711 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.048s user 0.043s sys 0.004s
I20240501 20:12:35.185855 17711 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:35.186216 17711 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:35.186458 17711 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:35.187474 17711 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:35.188627 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-eeaff69149ee4707beba512799b0d0df received from {username='slave'} at 127.28.136.73:37299
I20240501 20:12:35.188804 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-eeaff69149ee4707beba512799b0d0df on tablet eeaff69149ee4707beba512799b0d0df with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:35.293501 14563 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } }
I20240501 20:12:35.296929 14298 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:35.296947 14694 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:35.297578 17676 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:35.298049 17692 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:35.299417 14429 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:35.299551 14035 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:35.300158 17676 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:35.300684 17676 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:35.303323 14169 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:35.304270 13905 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:12:35.304965 17647 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:35.305521 17647 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:35.314738 17647 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:35.316351 13905 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:35.324231 14694 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:35.325129 13762 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:35.325556 14169 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:35.324532 14035 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:35.330250 14298 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:35.336494 14563 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:35.344540 14037 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:35.345561 17676 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:35.346750 14296 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:35.347509 17676 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:35.350069 14694 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:35.350716 17676 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:35.354843 13904 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:35.355465 14169 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:35.356185 17676 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:35.357306 17613 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:35.362077 14168 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:35.363794 14694 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:35.364032 14298 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:35.364452 13761 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:35.364971 13750 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 48770e5bebdf4d0b8766738665f7b560 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:12:35.367486 14431 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:12:35.368852 17692 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:12:35.369310 17645 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:12:35.370348 14431 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } }
I20240501 20:12:35.371309 13905 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:35.373992 14035 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:35.377296 14431 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
W20240501 20:12:35.418274 14498 consensus_peers.cc:485] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48770e5bebdf4d0b8766738665f7b560. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:35.448930 17711 ts_tablet_manager.cc:902] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:35.450188 17711 tablet_copy_client.cc:308] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:35.450946 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet 59fdeaff3d92460c8ca59eee1144b127 from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:37299)
I20240501 20:12:35.451129 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet 59fdeaff3d92460c8ca59eee1144b127 from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:37299: session id = 2a31af84f57c4ef18f0d98ff044229b6-59fdeaff3d92460c8ca59eee1144b127
I20240501 20:12:35.453374 14583 tablet_copy_source_session.cc:205] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:35.454815 17711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:35.459368 17711 tablet_copy_client.cc:783] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:35.459602 17711 tablet_copy_client.cc:647] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:35.461421 17711 tablet_copy_client.cc:523] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:35.464011 17711 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:35.495679 17711 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:35.496100 17711 tablet_bootstrap.cc:492] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:35.496330 17711 ts_tablet_manager.cc:1367] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20240501 20:12:35.497133 17711 raft_consensus.cc:348] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:35.497404 17711 raft_consensus.cc:729] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:35.497649 17711 consensus_queue.cc:260] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:35.498203 17711 ts_tablet_manager.cc:1398] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:12:35.499114 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-59fdeaff3d92460c8ca59eee1144b127 received from {username='slave'} at 127.28.136.73:37299
I20240501 20:12:35.499290 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-59fdeaff3d92460c8ca59eee1144b127 on tablet 59fdeaff3d92460c8ca59eee1144b127 with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:35.730407 16569 raft_consensus.cc:1203] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:35.830055 16569 raft_consensus.cc:1203] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:35.854074 17769 ts_tablet_manager.cc:902] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:35.855469 17769 tablet_copy_client.cc:308] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:35.863183 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet 48770e5bebdf4d0b8766738665f7b560 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:36771)
I20240501 20:12:35.863384 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet 48770e5bebdf4d0b8766738665f7b560 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:36771: session id = aeb70769b705493bb5196527311b142b-48770e5bebdf4d0b8766738665f7b560
I20240501 20:12:35.865455 14583 tablet_copy_source_session.cc:205] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:35.866729 17769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:35.876271 17769 tablet_copy_client.cc:783] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:35.876598 17769 tablet_copy_client.cc:647] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:35.878520 17769 tablet_copy_client.cc:523] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:35.883675 17769 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:35.926147 17769 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:35.926635 17769 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:35.926956 17769 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.043s user 0.036s sys 0.004s
I20240501 20:12:35.927841 17769 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:35.928153 17769 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:35.928373 17769 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:35.929297 17769 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 20:12:35.929960 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session aeb70769b705493bb5196527311b142b-48770e5bebdf4d0b8766738665f7b560 received from {username='slave'} at 127.28.136.72:36771
I20240501 20:12:35.930091 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session aeb70769b705493bb5196527311b142b-48770e5bebdf4d0b8766738665f7b560 on tablet 48770e5bebdf4d0b8766738665f7b560 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:36.233798 17647 raft_consensus.cc:1050] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:36.234927 17647 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:36.241745 13905 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.241755 16569 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.241816 14169 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.243001 17645 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.243600 14296 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.243587 17811 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.245545 17692 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.246062 14037 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.246752 17811 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.248579 17692 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.266625 14692 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.267872 14431 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.268570 17645 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.269445 17645 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.276322 17645 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.277238 14431 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.280421 14692 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.281050 14037 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.284305 13761 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:36.284446 14296 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.286273 14169 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.291049 16569 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.302585 13905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.309891 17647 raft_consensus.cc:1050] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:36.311098 17647 consensus_queue.cc:237] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:36.317085 14169 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.317939 17812 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.318373 14692 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.318490 16569 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.318590 13905 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.319236 17647 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.319416 14037 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.319471 14296 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.319693 17812 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.320053 17645 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.320353 17813 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.320695 14431 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.321267 17811 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.322314 17645 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.338801 17645 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.340620 16568 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.344225 14296 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.344391 14037 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.344825 14692 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.346241 13905 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.353196 14431 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.359532 14169 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.374948 13764 catalog_manager.cc:5484] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 5 to 6, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.398875 14563 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:36.401544 14562 consensus_queue.cc:237] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:36.405040 16569 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.405297 13905 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.405854 17692 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.406479 17692 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.407076 14035 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.408886 14431 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.409174 14169 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.410053 17812 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.411765 14692 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.412724 17647 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.413766 17812 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.416277 13905 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.416728 16304 raft_consensus.cc:1203] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:12:36.417068 14692 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.417498 14168 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.418504 17692 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.420114 16568 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.421149 17813 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.421945 17813 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.422868 14035 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.422570 17813 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.423427 17814 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.424316 14035 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.425961 17812 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.426108 14431 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.426607 17812 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.430941 17812 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.431299 16568 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.433051 13762 catalog_manager.cc:5484] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.436131 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eeaff69149ee4707beba512799b0d0df with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:36.437662 17812 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240501 20:12:36.437914 13905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.439074 13765 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:36.439810 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 59fdeaff3d92460c8ca59eee1144b127 with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:36.439069 14692 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.431517 14035 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.443645 14278 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaff69149ee4707beba512799b0d0df with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:36.444890 17840 tablet_replica.cc:330] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:36.445156 17840 raft_consensus.cc:2229] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:36.445602 17840 raft_consensus.cc:2258] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:36.446187 16568 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.447836 17840 ts_tablet_manager.cc:1875] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:36.451026 14431 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.452256 14169 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.456360 17840 ts_tablet_manager.cc:1888] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:36.456595 17840 log.cc:1196] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/59fdeaff3d92460c8ca59eee1144b127
I20240501 20:12:36.457866 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet 59fdeaff3d92460c8ca59eee1144b127 (table rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]) successfully deleted
I20240501 20:12:36.459618 13905 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.462255 14429 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.462266 14168 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.465859 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 59fdeaff3d92460c8ca59eee1144b127 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:36.469246 17840 tablet_replica.cc:330] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:36.469534 17840 raft_consensus.cc:2229] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:36.470532 17840 raft_consensus.cc:2258] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:36.472528 14692 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:36.473873 17840 ts_tablet_manager.cc:1875] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:36.479105 17840 ts_tablet_manager.cc:1888] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:36.480228 17840 log.cc:1196] T eeaff69149ee4707beba512799b0d0df P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/eeaff69149ee4707beba512799b0d0df
I20240501 20:12:36.481110 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet eeaff69149ee4707beba512799b0d0df (table rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]) successfully deleted
I20240501 20:12:36.931545 17813 raft_consensus.cc:1050] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:36.932586 17813 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:36.935632 14692 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.935891 16304 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.936111 14296 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.936625 17645 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.937052 17813 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.937474 14429 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.937590 14168 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.937824 17814 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.938369 17812 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.938689 17692 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.939193 14035 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.940008 17647 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.944365 13905 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:12:36.946157 17812 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:12:36.947049 17812 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.948015 16304 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.954344 14692 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.956609 13762 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:36.958608 14429 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.954638 14296 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.963238 14169 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.973481 14037 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.978812 13905 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.986114 14563 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:36.990650 16304 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.991086 14168 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.991159 14692 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.991619 17692 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.991879 14429 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.991880 17645 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.992061 14035 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.992267 17645 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.992517 17812 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.992709 17645 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.992847 13905 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:12:36.993402 17813 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:12:36.996637 17813 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.998270 13905 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:36.998759 16304 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:37.001379 14168 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:37.006428 14037 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:37.009507 13761 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b reported cstate change: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:37.011286 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48770e5bebdf4d0b8766738665f7b560 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:37.011175 14692 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:37.011960 14429 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:37.020416 14279 tablet_service.cc:1514] Processing DeleteTablet for tablet 48770e5bebdf4d0b8766738665f7b560 with delete_type TABLET_DATA_TOMBSTONED (TS 692bb14bda8046b9827f1091b0946088 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39618
I20240501 20:12:37.021232 17873 tablet_replica.cc:330] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: stopping tablet replica
I20240501 20:12:37.021456 17873 raft_consensus.cc:2229] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:37.022018 17873 raft_consensus.cc:2258] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:37.023356 17873 ts_tablet_manager.cc:1875] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:37.029247 17873 ts_tablet_manager.cc:1888] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:12:37.029381 17873 log.cc:1196] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-3/wal/wals/48770e5bebdf4d0b8766738665f7b560
I20240501 20:12:37.030088 13750 catalog_manager.cc:4830] TS 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): tablet 48770e5bebdf4d0b8766738665f7b560 (table rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]) successfully deleted
I20240501 20:12:37.494987 14147 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:37.513317 16416 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.519670 14674 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:37.531565 16284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.532020 16549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.548767 14279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.562115 14017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:37.582254 14542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:37.593664 14410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:37.673089 13884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:37.929498 14147 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:37.935833 16549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.938292 16416 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.964516 14279 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:37.981343 16284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:38.012784 14410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:38.014062 14674 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:38.026469 14542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:38.049845 13884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:38.120887 14017 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:38.218858 14429 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:38.222644 16435 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.222681 14692 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.223415 17793 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.223906 17978 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:12:38.230723 14367 consensus_peers.cc:485] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6213df9f24a4e63aec805f7dfacc75d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:38.232478 14563 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.233158 14035 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.233203 17793 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.233796 14168 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.234966 17879 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.235242 13905 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.235626 17793 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.236096 17879 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.257550 17978 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.258723 14692 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.260677 13905 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.266258 13757 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.266775 14563 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.268427 16435 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.293310 14168 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.312309 14035 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.340590 14692 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:38.350782 16568 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:12:38.351802 14629 consensus_peers.cc:485] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e21fb0e6bac46fbbd29bbd2f1bf43b4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:38.352048 14296 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.352025 17126 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.354416 14168 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.359164 17792 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.360569 17792 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.370209 14035 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.370942 14429 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.371083 17126 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.371708 17126 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.373044 13905 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.373026 17792 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.373682 17126 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.379609 14298 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.381515 16568 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.430718 13763 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.430291 13905 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.436244 14168 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.438782 14035 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.452535 14431 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.474491 14692 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:38.477571 14562 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:12:38.478256 14630 consensus_peers.cc:485] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af0f6a4c2e9c43509502a242b596d9da. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:38.478319 16568 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.478432 17998 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.479544 18014 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.479687 14035 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.479902 14168 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.480533 13905 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.480594 14429 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.481045 18014 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.481315 17998 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.483630 17792 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.484162 18014 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.495676 17998 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.498416 13905 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.498656 14431 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.503021 14168 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.503127 14562 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.505843 13760 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.508999 16568 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.512430 14035 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.532294 14694 consensus_queue.cc:237] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:38.535414 16435 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.535563 14035 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.535651 14168 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.536182 18005 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.536595 17126 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.537138 14562 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.537451 17792 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.538161 14298 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.540407 18005 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.541028 17792 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:12:38.551771 14630 consensus_peers.cc:485] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56c262e138874431a07f6d0bfcb516bf. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:38.552655 16434 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.554662 14037 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.557746 13763 catalog_manager.cc:5484] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 reported cstate change: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.550796 17998 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.560603 13905 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.560657 14168 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.561221 18014 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.564137 14562 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.570581 14296 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.583844 13905 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.603268 14562 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:38.606808 16304 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.607501 17969 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.620867 14168 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.622112 14694 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.622222 14429 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.623320 13905 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.623356 17969 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.623957 18043 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.624389 14037 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.624500 17968 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.625305 17968 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.626463 18043 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.630059 18042 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.630648 14037 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.637154 16304 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.637527 13904 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.641362 14694 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.643672 13762 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:12:38.644300 14497 consensus_peers.cc:485] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48770e5bebdf4d0b8766738665f7b560. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:38.651749 14169 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.672065 14429 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.683250 14562 consensus_queue.cc:237] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:38.686487 14296 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.686924 14694 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.687019 16568 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.687091 14168 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.687896 17968 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.688300 18059 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.688972 18043 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:12:38.689296 14498 consensus_peers.cc:485] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6335cd6db4474648bdf39486b7c9bb09. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:38.689579 18043 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.690295 13904 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.691566 18043 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.694232 14037 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.694918 17969 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.696162 17969 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.698305 13904 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.701455 14168 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.704003 14694 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.705494 13763 catalog_manager.cc:5484] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:38.709738 16568 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.713771 14296 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.722226 14037 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:38.734429 14037 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:38.738106 14168 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.738382 13904 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:12:38.738896 13974 consensus_peers.cc:485] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8892d6dce05140babdeeb6c90c366b3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:38.739028 17669 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.739719 17669 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.740401 14694 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.741822 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.746031 14296 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.747232 18074 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.747577 16304 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.748314 18074 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.752801 17669 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.757483 14562 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.760390 14296 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.761468 13763 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:38.772891 18081 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.774977 14168 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.779844 14694 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.782029 13904 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.789077 16304 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.805202 14037 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:38.822912 14694 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.823547 18083 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.823675 14296 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.824662 16304 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.825196 17669 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.825543 14563 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.825711 17669 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.827456 18083 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.827551 13905 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.828126 18083 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.830988 14168 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.843830 18082 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:12:38.870707 13973 consensus_peers.cc:485] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe1592bebd6b4a858987847446f15a16. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:38.873492 18091 ts_tablet_manager.cc:902] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:38.876188 18094 ts_tablet_manager.cc:902] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:38.878664 14562 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:38.889760 18082 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.893628 14692 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.898272 13763 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:38.905408 14296 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.908372 14169 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.914088 13901 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.917423 16304 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.919390 14562 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:38.941154 18091 tablet_copy_client.cc:308] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:32991
I20240501 20:12:38.953917 18094 tablet_copy_client.cc:308] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:38.954974 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:35471)
I20240501 20:12:38.955201 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:35471: session id = 1899220968384feebca3562bd7741a21-4e21fb0e6bac46fbbd29bbd2f1bf43b4
I20240501 20:12:38.958693 14714 tablet_copy_source_session.cc:205] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:38.960366 18094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:38.966112 14451 tablet_copy_service.cc:140] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Received BeginTabletCopySession request for tablet c6213df9f24a4e63aec805f7dfacc75d from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:37785)
I20240501 20:12:38.966307 14451 tablet_copy_service.cc:161] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Beginning new tablet copy session on tablet c6213df9f24a4e63aec805f7dfacc75d from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:37785: session id = 2a31af84f57c4ef18f0d98ff044229b6-c6213df9f24a4e63aec805f7dfacc75d
I20240501 20:12:38.969288 14451 tablet_copy_source_session.cc:205] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:38.970628 18094 tablet_copy_client.cc:783] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:38.970878 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:38.970947 18094 tablet_copy_client.cc:647] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:38.975112 18094 tablet_copy_client.cc:523] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:38.977574 18091 tablet_copy_client.cc:783] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:38.977809 18091 tablet_copy_client.cc:647] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:38.977921 18094 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:38.979808 18091 tablet_copy_client.cc:523] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:38.980480 14429 consensus_queue.cc:237] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:38.985586 16434 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.985946 14169 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:38.989441 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.990058 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:38.991406 18091 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:39.027809 14035 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.029808 18122 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.030426 14562 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.031599 14692 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.032611 18118 ts_tablet_manager.cc:902] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:39.037842 18123 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.043205 13904 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.044463 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.047492 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.051856 18123 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
W20240501 20:12:39.056978 14367 consensus_peers.cc:485] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ae91bfd01694bea81cda5ffc9e7bc77. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:39.065380 14035 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:39.068449 18118 tablet_copy_client.cc:308] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:39.069362 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet 56c262e138874431a07f6d0bfcb516bf from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:45703)
I20240501 20:12:39.069528 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet 56c262e138874431a07f6d0bfcb516bf from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:45703: session id = 2a31af84f57c4ef18f0d98ff044229b6-56c262e138874431a07f6d0bfcb516bf
I20240501 20:12:39.072372 14714 tablet_copy_source_session.cc:205] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:39.075618 18118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c262e138874431a07f6d0bfcb516bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:39.078439 13763 catalog_manager.cc:5484] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:39.079224 13904 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:39.082788 14169 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:39.082792 16434 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:39.092674 14692 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:39.094043 14562 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:39.102315 18118 tablet_copy_client.cc:783] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:39.103468 18118 tablet_copy_client.cc:647] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:39.106007 18118 tablet_copy_client.cc:523] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:39.109411 18118 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:39.112493 18094 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.112965 18094 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:39.113226 18094 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.135s user 0.071s sys 0.048s
I20240501 20:12:39.114192 18094 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:39.114538 18094 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:39.114837 18094 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:39.115532 18094 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:39.116832 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session 1899220968384feebca3562bd7741a21-4e21fb0e6bac46fbbd29bbd2f1bf43b4 received from {username='slave'} at 127.28.136.74:35471
I20240501 20:12:39.116986 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session 1899220968384feebca3562bd7741a21-4e21fb0e6bac46fbbd29bbd2f1bf43b4 on tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:39.118642 18142 ts_tablet_manager.cc:902] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:39.136893 18142 tablet_copy_client.cc:308] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:39.138236 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet af0f6a4c2e9c43509502a242b596d9da from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:33909)
I20240501 20:12:39.138432 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet af0f6a4c2e9c43509502a242b596d9da from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:33909: session id = aeb70769b705493bb5196527311b142b-af0f6a4c2e9c43509502a242b596d9da
I20240501 20:12:39.140307 18150 ts_tablet_manager.cc:902] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:39.141010 14714 tablet_copy_source_session.cc:205] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:39.142517 18142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:39.145658 14429 consensus_queue.cc:237] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.154175 16434 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.154408 14169 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.155077 18138 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.157162 18138 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.159487 13904 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.159636 14035 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.159802 14692 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.160284 18124 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:12:39.161648 14367 consensus_peers.cc:485] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39bedd9f3d3c4ee2890206cb43591675. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:39.161710 14296 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:39.162425 17879 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.162971 17879 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.164059 18138 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:39.184144 18091 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.184660 18091 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:39.185027 18091 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.194s user 0.049s sys 0.005s
I20240501 20:12:39.185899 18091 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.186244 18091 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:39.186470 18091 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.187125 18091 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:12:39.192616 18142 tablet_copy_client.cc:783] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:39.192921 18142 tablet_copy_client.cc:647] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:39.197870 17879 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:39.198547 16435 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:39.200855 18142 tablet_copy_client.cc:523] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:39.204272 14451 tablet_copy_service.cc:342] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-c6213df9f24a4e63aec805f7dfacc75d received from {username='slave'} at 127.28.136.73:37785
I20240501 20:12:39.204391 14451 tablet_copy_service.cc:433] P 3b9f58142e4a4ccd9d0cbafc97d459c7: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-c6213df9f24a4e63aec805f7dfacc75d on tablet c6213df9f24a4e63aec805f7dfacc75d with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:39.204701 13763 catalog_manager.cc:5484] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 reported cstate change: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:39.206431 14694 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:39.210928 14169 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:39.213263 14035 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:39.218014 18118 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.218564 18118 tablet_bootstrap.cc:492] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:39.218920 18118 ts_tablet_manager.cc:1367] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.110s user 0.048s sys 0.011s
I20240501 20:12:39.219846 18118 raft_consensus.cc:348] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.221922 18118 raft_consensus.cc:729] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:39.224387 18094 ts_tablet_manager.cc:902] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:39.231802 13904 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:39.231984 14296 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:39.236913 18142 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:39.222229 18118 consensus_queue.cc:260] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.248252 18118 ts_tablet_manager.cc:1398] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240501 20:12:39.249615 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-56c262e138874431a07f6d0bfcb516bf received from {username='slave'} at 127.28.136.73:45703
I20240501 20:12:39.249755 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-56c262e138874431a07f6d0bfcb516bf on tablet 56c262e138874431a07f6d0bfcb516bf with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:39.252926 18150 tablet_copy_client.cc:308] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:39.253712 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet 8892d6dce05140babdeeb6c90c366b3b from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:45763)
I20240501 20:12:39.253928 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet 8892d6dce05140babdeeb6c90c366b3b from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:45763: session id = 2a31af84f57c4ef18f0d98ff044229b6-8892d6dce05140babdeeb6c90c366b3b
I20240501 20:12:39.256722 14058 tablet_copy_source_session.cc:205] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:39.259835 18150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:39.269191 18150 tablet_copy_client.cc:783] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:39.269461 18150 tablet_copy_client.cc:647] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:39.271929 18150 tablet_copy_client.cc:523] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:39.276317 18150 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:39.287736 18170 ts_tablet_manager.cc:902] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:39.289062 18094 tablet_copy_client.cc:308] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:39.313985 18176 ts_tablet_manager.cc:902] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:39.329991 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet 48770e5bebdf4d0b8766738665f7b560 from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:34649)
I20240501 20:12:39.330175 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet 48770e5bebdf4d0b8766738665f7b560 from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:34649: session id = 1899220968384feebca3562bd7741a21-48770e5bebdf4d0b8766738665f7b560
I20240501 20:12:39.333258 14583 tablet_copy_source_session.cc:205] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:39.333727 18170 tablet_copy_client.cc:308] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:39.334911 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet 6335cd6db4474648bdf39486b7c9bb09 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:36771)
I20240501 20:12:39.335665 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet 6335cd6db4474648bdf39486b7c9bb09 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:36771: session id = aeb70769b705493bb5196527311b142b-6335cd6db4474648bdf39486b7c9bb09
I20240501 20:12:39.336895 18094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:39.339232 14583 tablet_copy_source_session.cc:205] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:39.345070 18170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6335cd6db4474648bdf39486b7c9bb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:39.346458 18094 tablet_copy_client.cc:783] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:39.346833 18094 tablet_copy_client.cc:647] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:39.348776 18094 tablet_copy_client.cc:523] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:39.349962 18176 tablet_copy_client.cc:308] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:39.358372 18170 tablet_copy_client.cc:783] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:39.363258 18094 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:39.373713 18170 tablet_copy_client.cc:647] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:39.380232 18170 tablet_copy_client.cc:523] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:39.394838 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet fe1592bebd6b4a858987847446f15a16 from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:35733)
I20240501 20:12:39.395069 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet fe1592bebd6b4a858987847446f15a16 from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:35733: session id = 1899220968384feebca3562bd7741a21-fe1592bebd6b4a858987847446f15a16
I20240501 20:12:39.400424 14058 tablet_copy_source_session.cc:205] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:39.403101 18176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:39.404196 18170 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:39.404302 16568 raft_consensus.cc:1203] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.413367 18176 tablet_copy_client.cc:783] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:39.413479 16434 raft_consensus.cc:1203] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.413975 18192 ts_tablet_manager.cc:902] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:39.420604 18176 tablet_copy_client.cc:647] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:39.423110 18176 tablet_copy_client.cc:523] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:39.433706 18176 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:39.477551 18192 tablet_copy_client.cc:308] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:32991
I20240501 20:12:39.487434 18150 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.487910 18150 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:39.488243 18150 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.212s user 0.052s sys 0.002s
I20240501 20:12:39.496471 16568 raft_consensus.cc:1203] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.507160 18150 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.507581 18150 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:39.507830 18150 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.508487 18150 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240501 20:12:39.509195 18094 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.509708 18094 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:39.509992 18094 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.147s user 0.043s sys 0.031s
I20240501 20:12:39.511240 18094 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:39.511564 18094 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:39.511792 18094 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:39.512418 18094 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:39.516849 14451 tablet_copy_service.cc:140] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Received BeginTabletCopySession request for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:45793)
I20240501 20:12:39.517022 14451 tablet_copy_service.cc:161] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Beginning new tablet copy session on tablet 9ae91bfd01694bea81cda5ffc9e7bc77 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:45793: session id = aeb70769b705493bb5196527311b142b-9ae91bfd01694bea81cda5ffc9e7bc77
I20240501 20:12:39.519964 14451 tablet_copy_source_session.cc:205] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:39.520529 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-8892d6dce05140babdeeb6c90c366b3b received from {username='slave'} at 127.28.136.73:45763
I20240501 20:12:39.520714 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-8892d6dce05140babdeeb6c90c366b3b on tablet 8892d6dce05140babdeeb6c90c366b3b with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:39.521790 18192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae91bfd01694bea81cda5ffc9e7bc77. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:12:39.523605 14495 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:44849, user_credentials={real_user=slave}} blocked reactor thread for 56091.1us
I20240501 20:12:39.524497 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session 1899220968384feebca3562bd7741a21-48770e5bebdf4d0b8766738665f7b560 received from {username='slave'} at 127.28.136.74:34649
I20240501 20:12:39.524681 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session 1899220968384feebca3562bd7741a21-48770e5bebdf4d0b8766738665f7b560 on tablet 48770e5bebdf4d0b8766738665f7b560 with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:39.535142 18192 tablet_copy_client.cc:783] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:39.535447 18192 tablet_copy_client.cc:647] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:39.538722 18170 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.539234 18192 tablet_copy_client.cc:523] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:39.540058 18170 tablet_bootstrap.cc:492] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:39.540424 18170 ts_tablet_manager.cc:1367] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.137s user 0.058s sys 0.004s
I20240501 20:12:39.541414 18170 raft_consensus.cc:348] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:39.541822 18170 raft_consensus.cc:729] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:39.542129 18170 consensus_queue.cc:260] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:39.542881 18192 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:39.547577 18170 ts_tablet_manager.cc:1398] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:12:39.555207 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session aeb70769b705493bb5196527311b142b-6335cd6db4474648bdf39486b7c9bb09 received from {username='slave'} at 127.28.136.72:36771
I20240501 20:12:39.555392 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session aeb70769b705493bb5196527311b142b-6335cd6db4474648bdf39486b7c9bb09 on tablet 6335cd6db4474648bdf39486b7c9bb09 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:39.569622 18176 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.570117 18176 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:39.570428 18176 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.137s user 0.051s sys 0.004s
I20240501 20:12:39.571301 18176 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:39.571650 18176 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:39.571889 18176 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:39.572664 18176 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:39.573769 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session 1899220968384feebca3562bd7741a21-fe1592bebd6b4a858987847446f15a16 received from {username='slave'} at 127.28.136.74:35733
I20240501 20:12:39.573920 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session 1899220968384feebca3562bd7741a21-fe1592bebd6b4a858987847446f15a16 on tablet fe1592bebd6b4a858987847446f15a16 with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:39.582573 18150 ts_tablet_manager.cc:902] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:39.583099 18142 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.583952 18142 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:39.584355 18142 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.348s user 0.054s sys 0.007s
I20240501 20:12:39.585383 18142 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:39.585964 18142 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:39.586268 18142 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:39.587195 18142 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:39.588022 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session aeb70769b705493bb5196527311b142b-af0f6a4c2e9c43509502a242b596d9da received from {username='slave'} at 127.28.136.72:33909
I20240501 20:12:39.588146 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session aeb70769b705493bb5196527311b142b-af0f6a4c2e9c43509502a242b596d9da on tablet af0f6a4c2e9c43509502a242b596d9da with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:39.602417 18150 tablet_copy_client.cc:308] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:32991
I20240501 20:12:39.603253 14451 tablet_copy_service.cc:140] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Received BeginTabletCopySession request for tablet 39bedd9f3d3c4ee2890206cb43591675 from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:37785)
I20240501 20:12:39.603433 14451 tablet_copy_service.cc:161] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Beginning new tablet copy session on tablet 39bedd9f3d3c4ee2890206cb43591675 from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:37785: session id = 2a31af84f57c4ef18f0d98ff044229b6-39bedd9f3d3c4ee2890206cb43591675
I20240501 20:12:39.605829 14451 tablet_copy_source_session.cc:205] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:39.606117 18192 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.607249 18150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39bedd9f3d3c4ee2890206cb43591675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:39.607800 18192 tablet_bootstrap.cc:492] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:39.608043 18192 ts_tablet_manager.cc:1367] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.065s user 0.045s sys 0.008s
I20240501 20:12:39.608640 18192 raft_consensus.cc:348] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:39.608953 18192 raft_consensus.cc:729] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:39.609161 18192 consensus_queue.cc:260] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:39.609787 18192 ts_tablet_manager.cc:1398] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:12:39.613168 18150 tablet_copy_client.cc:783] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:39.613384 18150 tablet_copy_client.cc:647] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:39.614462 14451 tablet_copy_service.cc:342] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Request end of tablet copy session aeb70769b705493bb5196527311b142b-9ae91bfd01694bea81cda5ffc9e7bc77 received from {username='slave'} at 127.28.136.72:45793
I20240501 20:12:39.614615 14451 tablet_copy_service.cc:433] P 3b9f58142e4a4ccd9d0cbafc97d459c7: ending tablet copy session aeb70769b705493bb5196527311b142b-9ae91bfd01694bea81cda5ffc9e7bc77 on tablet 9ae91bfd01694bea81cda5ffc9e7bc77 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:39.615275 18150 tablet_copy_client.cc:523] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:39.618335 18150 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:39.618773 16568 raft_consensus.cc:1203] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.662307 16304 raft_consensus.cc:1203] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.667865 18150 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:39.668332 18150 tablet_bootstrap.cc:492] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:39.668591 18150 ts_tablet_manager.cc:1367] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.050s user 0.045s sys 0.002s
I20240501 20:12:39.669340 18150 raft_consensus.cc:348] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.669615 18150 raft_consensus.cc:729] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:39.669972 18150 consensus_queue.cc:260] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:39.670589 18150 ts_tablet_manager.cc:1398] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 20:12:39.671296 14451 tablet_copy_service.cc:342] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-39bedd9f3d3c4ee2890206cb43591675 received from {username='slave'} at 127.28.136.73:37785
I20240501 20:12:39.671427 14451 tablet_copy_service.cc:433] P 3b9f58142e4a4ccd9d0cbafc97d459c7: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-39bedd9f3d3c4ee2890206cb43591675 on tablet 39bedd9f3d3c4ee2890206cb43591675 with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:39.690101 16434 raft_consensus.cc:1203] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.696213 16304 raft_consensus.cc:1203] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.762097 16304 raft_consensus.cc:1203] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.858500 16434 raft_consensus.cc:1203] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:39.971608 17126 raft_consensus.cc:1050] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:39.971668 17879 raft_consensus.cc:1050] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:39.972481 17126 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:39.972510 17879 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:39.975575 16434 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:39.975677 16568 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:39.976140 14298 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:39.976902 17998 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:39.980454 18027 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:39.983584 17998 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:39.999312 16435 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.000207 18138 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.002923 16568 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.003741 18138 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.014094 18296 raft_consensus.cc:1050] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:40.017110 14692 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.015605 18296 consensus_queue.cc:237] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:40.018393 18138 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.024602 14035 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.024725 13904 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.025018 14169 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.025255 18138 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.025632 18301 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.028242 14562 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.031577 17879 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.031891 18138 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.050424 14168 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.051153 18027 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.052650 14429 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.054716 18027 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.065435 18138 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.066238 13901 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.067385 14166 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.071463 16435 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.071558 16568 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.073177 13902 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.073869 18027 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.075752 14037 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.077840 13759 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:40.081162 18293 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.089998 14692 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.097811 18027 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.105739 14563 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.114809 18206 raft_consensus.cc:1050] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:40.116014 18206 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:40.120157 14036 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.126662 16435 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.128216 18293 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.133076 16434 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.141021 16304 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.140830 14037 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.141642 14298 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.143339 14169 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.144001 18060 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.146981 14692 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.147866 18203 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.148478 18203 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.155340 14431 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.157084 18060 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.158124 14035 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.158700 18060 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.160059 13758 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.161846 16568 raft_consensus.cc:1203] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:40.163122 13904 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.167287 14563 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.168135 18323 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.171905 16434 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.172631 18060 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.173384 18060 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.197321 18060 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.205353 18203 raft_consensus.cc:1050] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:40.204757 16304 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.204855 14035 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.206504 18203 consensus_queue.cc:237] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:40.209290 16434 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.211437 13756 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b reported cstate change: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.212826 14694 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.214088 16568 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.214646 16304 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.214748 18339 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.219003 18060 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.220840 18206 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.219949 14431 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.221467 18123 raft_consensus.cc:1050] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:40.222749 18123 consensus_queue.cc:237] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:40.237079 18083 raft_consensus.cc:1050] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:40.238435 18083 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:40.238857 16435 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.237638 14168 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.239552 18138 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.242504 14166 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.233918 14429 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.243364 18138 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.248296 14691 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.251528 14167 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.254271 13902 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.254498 16568 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.255308 18082 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.255935 18082 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.258209 16303 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.258816 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:12:40.262472 14498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.72:39739, user_credentials={real_user=slave}} blocked reactor thread for 47441.5us
I20240501 20:12:40.265946 14298 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.266682 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.271593 13906 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.272325 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.274649 14562 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.274899 14035 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.275210 18138 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.275723 18138 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.275902 14691 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.276336 14562 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.276582 18138 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.278043 18083 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.282258 14167 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.283010 18083 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.284387 14694 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.287998 16304 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.288004 16619 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.288697 18123 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.303949 16619 raft_consensus.cc:1050] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:40.305217 16619 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:40.307852 14430 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:40.312556 14037 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.313484 18301 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.317204 16567 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.318770 13901 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.319399 14169 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.319882 18301 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.320127 18206 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.320369 14037 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.320967 18206 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.324124 16435 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.324334 14692 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.323805 14298 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.325217 18301 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.325511 18339 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.325789 18301 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.327056 13901 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.327790 14563 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.328413 18138 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.329483 18138 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.330046 14693 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.331872 16434 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.333500 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6213df9f24a4e63aec805f7dfacc75d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:40.336400 14561 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.345108 16567 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.346194 13904 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.348227 17879 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.349192 14692 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:40.348556 13761 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.352097 18123 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.352224 18060 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.353989 14036 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.356000 16567 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.355927 14561 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:40.357424 14038 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.358685 16567 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.360591 14034 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.363049 18082 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.363689 14693 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.364940 14431 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.365579 18339 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.365214 13901 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.366645 18339 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.367154 18203 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.368381 16435 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.368999 18339 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.369905 14147 tablet_service.cc:1514] Processing DeleteTablet for tablet c6213df9f24a4e63aec805f7dfacc75d with delete_type TABLET_DATA_TOMBSTONED (TS 0e898d8ee23441908d1593c561b368ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44418
I20240501 20:12:40.378522 18206 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.381163 14038 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.382851 16436 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.383224 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48770e5bebdf4d0b8766738665f7b560 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:40.383442 18319 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.386687 17126 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.389014 14693 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.389848 14431 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.394553 16436 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.395591 13761 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.401970 14148 tablet_service.cc:1514] Processing DeleteTablet for tablet 48770e5bebdf4d0b8766738665f7b560 with delete_type TABLET_DATA_TOMBSTONED (TS 0e898d8ee23441908d1593c561b368ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44418
I20240501 20:12:40.368536 13904 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.410089 17126 raft_consensus.cc:1050] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:40.410508 16569 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.410725 16303 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.411245 17126 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:40.415781 13902 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.416286 14296 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.416903 18324 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.417624 14169 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.417675 18060 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.418291 18324 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.418303 18060 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.420919 18321 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.422742 13902 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.425834 13903 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.425812 18324 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.428181 14296 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.427347 16303 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.431699 13761 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:40.434563 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:40.435523 14169 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.439245 14167 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.440907 16434 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.442049 16436 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.443436 14561 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.444862 13764 catalog_manager.cc:5484] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:40.445065 14167 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.447701 14166 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.453159 18321 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240501 20:12:40.455643 14296 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.457950 18321 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.458101 16567 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.460557 18060 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.463038 14036 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.463894 13756 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
W20240501 20:12:40.470209 14630 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.73:39773, user_credentials={real_user=slave}} blocked reactor thread for 44490.7us
I20240501 20:12:40.471673 16569 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.474491 14562 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.475262 18027 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.475711 14429 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.478266 14296 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.482916 18296 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.483471 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with delete_type TABLET_DATA_TOMBSTONED (TS c3224a319e2f4cb8a84bf9528209a9fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56540
I20240501 20:12:40.484570 18027 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.486209 14430 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.486950 14167 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.487758 18027 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.488194 16302 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.488377 18027 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.490687 14429 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.491292 16566 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.479465 13906 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.492061 16303 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.493258 13902 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.495262 13904 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.496467 18324 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.503522 18321 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.505656 18324 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.508867 14297 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.513191 18383 tablet_replica.cc:330] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: stopping tablet replica
I20240501 20:12:40.513792 18383 raft_consensus.cc:2229] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:40.514277 18383 raft_consensus.cc:2258] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:40.516214 18383 ts_tablet_manager.cc:1875] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:40.516968 14036 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.517192 14038 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.517140 16303 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.517452 13902 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.523406 18296 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.524184 18319 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.527297 18370 tablet_replica.cc:330] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: stopping tablet replica
I20240501 20:12:40.527323 18383 ts_tablet_manager.cc:1888] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:40.527942 18370 raft_consensus.cc:2229] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:40.528429 18370 raft_consensus.cc:2258] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:40.530530 18370 ts_tablet_manager.cc:1875] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:40.543210 13903 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.545233 18296 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.549468 16302 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.553997 17126 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.560621 14562 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.562589 16302 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.566893 16569 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.565799 13764 catalog_manager.cc:5484] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b reported cstate change: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.569221 14694 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.573073 18324 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.575031 14429 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.578681 14692 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.582062 16567 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.585191 14561 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.592258 14693 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.593078 14430 consensus_queue.cc:237] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:40.593152 16568 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.594049 14562 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.596241 13761 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:40.576154 14691 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.599375 16304 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.598500 13757 catalog_manager.cc:5484] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.607236 18383 log.cc:1196] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/4e21fb0e6bac46fbbd29bbd2f1bf43b4
I20240501 20:12:40.607302 14296 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.608139 16304 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.593398 18372 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.608846 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.609431 14562 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.609987 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.612075 18368 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.613847 18371 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.591970 14560 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.618821 13904 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.619437 13906 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.618999 14295 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.619992 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.591105 14038 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.622975 17879 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.585357 14036 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.626190 13905 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.630997 16302 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.633134 14562 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.594738 14165 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.640822 18083 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.635764 13904 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.609428 13749 catalog_manager.cc:4830] TS c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 (table rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]) successfully deleted
I20240501 20:12:40.627941 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:40.645354 13756 catalog_manager.cc:5484] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.647075 16433 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.649286 16435 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.650684 18138 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.651130 16432 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.651762 14165 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.594604 14167 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.659116 16432 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.617348 14166 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.632766 14169 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.662217 18138 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.666286 14166 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.678220 18370 ts_tablet_manager.cc:1888] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:40.678407 18370 log.cc:1196] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/c6213df9f24a4e63aec805f7dfacc75d
I20240501 20:12:40.679435 18370 tablet_replica.cc:330] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: stopping tablet replica
I20240501 20:12:40.679661 18370 raft_consensus.cc:2229] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:40.679773 13751 catalog_manager.cc:4830] TS 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): tablet c6213df9f24a4e63aec805f7dfacc75d (table rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]) successfully deleted
I20240501 20:12:40.679927 18370 raft_consensus.cc:2258] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:40.681067 14692 raft_consensus.cc:1261] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.681887 17879 consensus_queue.cc:1035] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.682480 13906 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.683753 14691 raft_consensus.cc:2932] T 9ae91bfd01694bea81cda5ffc9e7bc77 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.687968 18371 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:12:40.691498 13973 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:38579, user_credentials={real_user=slave}} blocked reactor thread for 50861.7us
I20240501 20:12:40.691843 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae91bfd01694bea81cda5ffc9e7bc77 with delete_type TABLET_DATA_TOMBSTONED (TS c3224a319e2f4cb8a84bf9528209a9fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56540
I20240501 20:12:40.692513 18368 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.693060 18368 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.695933 14560 consensus_queue.cc:237] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:40.702342 14038 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:40.707587 14562 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.708213 18082 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.709877 14693 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.710691 18082 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.712031 16567 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.713547 14691 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.714411 14295 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.715579 18206 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.716141 18206 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:12:40.687933 13974 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:42939, user_credentials={real_user=slave}} blocked reactor thread for 48155us
I20240501 20:12:40.718312 16300 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.721161 14298 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.722013 18338 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.722772 16567 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.723462 16304 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.723603 18083 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.724020 18338 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.726540 18338 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.727476 18371 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.727401 16300 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.728596 18371 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.730967 14691 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.732612 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6335cd6db4474648bdf39486b7c9bb09 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:40.734151 16567 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.735810 14694 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:40.736387 14295 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.739670 14560 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.739926 14429 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.740358 17126 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.741221 17126 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.743901 13903 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.744320 13756 catalog_manager.cc:5484] T 6335cd6db4474648bdf39486b7c9bb09 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.746241 16302 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.745817 18082 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.749924 17126 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.751299 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet 6335cd6db4474648bdf39486b7c9bb09 with delete_type TABLET_DATA_TOMBSTONED (TS c3224a319e2f4cb8a84bf9528209a9fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56540
I20240501 20:12:40.752153 14038 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.751614 14298 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.752950 18338 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.753026 17126 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.754408 13749 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8892d6dce05140babdeeb6c90c366b3b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:40.756479 16567 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.756376 17126 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.757395 14562 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.756942 13903 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.761544 16300 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.763289 17879 raft_consensus.cc:1050] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:40.763592 14038 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.764554 17879 consensus_queue.cc:237] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:40.765118 18383 tablet_replica.cc:330] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: stopping tablet replica
I20240501 20:12:40.765360 18383 raft_consensus.cc:2229] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:40.765656 18383 raft_consensus.cc:2258] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:40.765367 16567 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.766744 18383 ts_tablet_manager.cc:1875] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:40.768002 14429 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.771801 13765 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.772756 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af0f6a4c2e9c43509502a242b596d9da with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:40.774441 13904 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.776517 14147 tablet_service.cc:1514] Processing DeleteTablet for tablet 8892d6dce05140babdeeb6c90c366b3b with delete_type TABLET_DATA_TOMBSTONED (TS 0e898d8ee23441908d1593c561b368ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44418
I20240501 20:12:40.777938 14165 raft_consensus.cc:1261] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.779323 18392 consensus_queue.cc:1035] T 6335cd6db4474648bdf39486b7c9bb09 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.775228 14560 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.782120 13756 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.782020 14167 raft_consensus.cc:2932] T 6335cd6db4474648bdf39486b7c9bb09 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.786454 14147 tablet_service.cc:1514] Processing DeleteTablet for tablet af0f6a4c2e9c43509502a242b596d9da with delete_type TABLET_DATA_TOMBSTONED (TS 0e898d8ee23441908d1593c561b368ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44418
I20240501 20:12:40.793017 14693 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.799387 14691 consensus_queue.cc:237] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:40.802141 18324 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.804960 16433 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.805773 14165 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.806450 18296 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.806932 18296 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.809069 14298 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.810300 18296 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.811264 16567 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.813017 18371 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.813620 18324 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.819068 14165 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.820142 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56c262e138874431a07f6d0bfcb516bf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:40.826164 14166 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.826326 13756 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.827819 16568 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.829298 16433 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.830814 18317 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.832427 13756 catalog_manager.cc:5484] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:40.832628 16566 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.834986 14295 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.833071 16432 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.837078 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet 56c262e138874431a07f6d0bfcb516bf with delete_type TABLET_DATA_TOMBSTONED (TS c3224a319e2f4cb8a84bf9528209a9fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56540
I20240501 20:12:40.837231 16303 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.842773 14296 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
W20240501 20:12:40.843218 13839 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:44849, user_credentials={real_user=slave}} blocked reactor thread for 149526us
I20240501 20:12:40.844972 13901 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.845979 18371 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.845856 14691 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.847944 13901 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.824532 14561 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.861171 13905 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.812942 14562 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.877300 14038 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.877725 13905 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.878141 18138 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.878883 18123 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.880210 16568 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.881208 14298 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.882010 18367 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.883273 18123 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.884872 13903 raft_consensus.cc:1261] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:40.889155 13902 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:40.894341 18370 ts_tablet_manager.cc:1875] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:40.895824 16433 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.897199 16302 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.898226 17879 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:12:40.908075 14629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:38579, user_credentials={real_user=slave}} blocked reactor thread for 48252.2us
I20240501 20:12:40.909257 14694 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.910239 18324 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.911978 18367 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.914743 14166 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:40.916751 18367 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:40.922299 18383 ts_tablet_manager.cc:1888] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:40.927414 18123 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.927809 16568 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.933913 13905 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.935333 14298 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.937942 13764 catalog_manager.cc:5484] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:40.942188 14560 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.936286 14038 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.948899 18383 log.cc:1196] T 9ae91bfd01694bea81cda5ffc9e7bc77 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/9ae91bfd01694bea81cda5ffc9e7bc77
I20240501 20:12:40.949890 18383 tablet_replica.cc:330] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: stopping tablet replica
I20240501 20:12:40.949944 13749 catalog_manager.cc:4830] TS c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): tablet 9ae91bfd01694bea81cda5ffc9e7bc77 (table rebalance_test_table_3 [id=681a9c86d6f64d72ac54c31f33a78ab8]) successfully deleted
I20240501 20:12:40.950238 18383 raft_consensus.cc:2229] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:40.950532 18383 raft_consensus.cc:2258] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:40.951699 18383 ts_tablet_manager.cc:1875] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:40.958163 18324 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:12:40.958822 14630 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.65:42939, user_credentials={real_user=slave}} blocked reactor thread for 94726.5us
I20240501 20:12:40.959281 18324 consensus_queue.cc:1035] T 56c262e138874431a07f6d0bfcb516bf P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:40.960893 18370 ts_tablet_manager.cc:1888] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:40.963016 18370 log.cc:1196] T 48770e5bebdf4d0b8766738665f7b560 P 0e898d8ee23441908d1593c561b368ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/48770e5bebdf4d0b8766738665f7b560
I20240501 20:12:40.965543 13751 catalog_manager.cc:4830] TS 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): tablet 48770e5bebdf4d0b8766738665f7b560 (table rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]) successfully deleted
I20240501 20:12:40.966065 18370 tablet_replica.cc:330] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: stopping tablet replica
I20240501 20:12:40.966513 18370 raft_consensus.cc:2229] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:40.966764 13905 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:40.968995 18370 raft_consensus.cc:2258] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:40.970073 18370 ts_tablet_manager.cc:1875] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:40.981436 18383 ts_tablet_manager.cc:1888] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:40.984967 13905 raft_consensus.cc:2932] T 56c262e138874431a07f6d0bfcb516bf P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:40.996313 18383 log.cc:1196] T 6335cd6db4474648bdf39486b7c9bb09 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/6335cd6db4474648bdf39486b7c9bb09
I20240501 20:12:40.997411 13749 catalog_manager.cc:4830] TS c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): tablet 6335cd6db4474648bdf39486b7c9bb09 (table rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]) successfully deleted
I20240501 20:12:40.999634 18383 tablet_replica.cc:330] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: stopping tablet replica
I20240501 20:12:40.999930 18383 raft_consensus.cc:2229] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:41.000286 18383 raft_consensus.cc:2258] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:41.002261 18383 ts_tablet_manager.cc:1875] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:41.005098 18370 ts_tablet_manager.cc:1888] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:41.012878 18370 log.cc:1196] T 8892d6dce05140babdeeb6c90c366b3b P 0e898d8ee23441908d1593c561b368ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/8892d6dce05140babdeeb6c90c366b3b
I20240501 20:12:41.013854 13751 catalog_manager.cc:4830] TS 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): tablet 8892d6dce05140babdeeb6c90c366b3b (table rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]) successfully deleted
I20240501 20:12:41.019165 18370 tablet_replica.cc:330] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: stopping tablet replica
I20240501 20:12:41.019408 18370 raft_consensus.cc:2229] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:41.019675 18370 raft_consensus.cc:2258] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:41.021011 18370 ts_tablet_manager.cc:1875] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:41.028673 18370 ts_tablet_manager.cc:1888] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:41.028961 18370 log.cc:1196] T af0f6a4c2e9c43509502a242b596d9da P 0e898d8ee23441908d1593c561b368ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/af0f6a4c2e9c43509502a242b596d9da
I20240501 20:12:41.029960 13751 catalog_manager.cc:4830] TS 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): tablet af0f6a4c2e9c43509502a242b596d9da (table rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]) successfully deleted
I20240501 20:12:41.032995 18383 ts_tablet_manager.cc:1888] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:41.034070 18383 log.cc:1196] T 56c262e138874431a07f6d0bfcb516bf P c3224a319e2f4cb8a84bf9528209a9fb: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/56c262e138874431a07f6d0bfcb516bf
I20240501 20:12:41.035037 13749 catalog_manager.cc:4830] TS c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): tablet 56c262e138874431a07f6d0bfcb516bf (table rebalance_test_table_4 [id=6e580c381169413f9fbb7ab112481fef]) successfully deleted
I20240501 20:12:41.041220 16432 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.048162 14694 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.050232 14430 consensus_queue.cc:237] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:41.053969 16567 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:41.054471 14296 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:41.055050 18367 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.055438 18301 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.059608 13905 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:41.060111 14038 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:41.060220 18301 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.062423 18367 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.065241 16433 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:41.064625 18123 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.068677 18367 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.069450 13905 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.069592 14038 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.073211 14693 raft_consensus.cc:1261] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:41.073730 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39bedd9f3d3c4ee2890206cb43591675 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:41.074651 16567 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.077231 13764 catalog_manager.cc:5484] T 39bedd9f3d3c4ee2890206cb43591675 P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.077482 16432 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.090018 14147 tablet_service.cc:1514] Processing DeleteTablet for tablet 39bedd9f3d3c4ee2890206cb43591675 with delete_type TABLET_DATA_TOMBSTONED (TS 0e898d8ee23441908d1593c561b368ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44418
I20240501 20:12:41.089675 14296 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.090636 18370 tablet_replica.cc:330] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: stopping tablet replica
I20240501 20:12:41.090855 18370 raft_consensus.cc:2229] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:41.091230 18370 pending_rounds.cc:62] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: Trying to abort 1 pending ops.
I20240501 20:12:41.091325 18370 pending_rounds.cc:68] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7022978501688430592 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "39bedd9f3d3c4ee2890206cb43591675" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } } }
I20240501 20:12:41.091905 18370 raft_consensus.cc:2894] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20240501 20:12:41.092027 18370 raft_consensus.cc:2866] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:12:41.092128 18370 raft_consensus.cc:2258] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:41.094830 17879 consensus_queue.cc:1035] T 39bedd9f3d3c4ee2890206cb43591675 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.096896 14694 raft_consensus.cc:2932] T 39bedd9f3d3c4ee2890206cb43591675 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:41.101027 18370 ts_tablet_manager.cc:1875] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:41.106765 18370 ts_tablet_manager.cc:1888] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:41.106886 18370 log.cc:1196] T 39bedd9f3d3c4ee2890206cb43591675 P 0e898d8ee23441908d1593c561b368ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/39bedd9f3d3c4ee2890206cb43591675
I20240501 20:12:41.107631 13751 catalog_manager.cc:4830] TS 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): tablet 39bedd9f3d3c4ee2890206cb43591675 (table rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]) successfully deleted
I20240501 20:12:41.147550 14038 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16"
dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36084
I20240501 20:12:41.147801 14038 raft_consensus.cc:595] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Received request to transfer leadership
I20240501 20:12:41.211200 14694 consensus_queue.cc:237] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:41.214473 16302 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.214522 14169 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.214773 14430 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.215345 14560 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.215432 18319 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.215919 18317 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.216250 18319 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.216811 14038 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.217408 18319 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.218179 13902 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 20:12:41.218276 14630 consensus_peers.cc:485] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77ef10335c1642818694b42ab006a66f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:41.218672 18317 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.219131 18317 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.224337 18324 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:41.225021 13902 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:41.230381 14169 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:41.235100 13761 catalog_manager.cc:5484] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:41.240033 14038 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:41.249465 16302 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:41.251226 14560 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:41.258252 18371 raft_consensus.cc:979] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: : Instructing follower 692bb14bda8046b9827f1091b0946088 to start an election
I20240501 20:12:41.258417 18371 raft_consensus.cc:1067] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Signalling peer 692bb14bda8046b9827f1091b0946088 to start an election
I20240501 20:12:41.260185 14296 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16"
dest_uuid: "692bb14bda8046b9827f1091b0946088"
from {username='slave'} at 127.28.136.66:34685
I20240501 20:12:41.260476 14296 raft_consensus.cc:482] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:12:41.260607 14296 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:41.263731 14296 raft_consensus.cc:504] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:41.269606 13905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c12d6cddb7f242b6af650a10bbc8eb69"
I20240501 20:12:41.269846 13905 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:41.270836 14038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c3224a319e2f4cb8a84bf9528209a9fb"
I20240501 20:12:41.271062 14038 raft_consensus.cc:3032] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:12:41.271157 14038 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Running, Role: LEADER
I20240501 20:12:41.271514 14038 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:41.272203 14038 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:41.272719 13905 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 2.
I20240501 20:12:41.275130 14038 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 2.
I20240501 20:12:41.277361 14430 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:41.280972 14169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "0e898d8ee23441908d1593c561b368ad"
I20240501 20:12:41.281271 14169 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:41.284160 14169 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 2.
I20240501 20:12:41.298950 14560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a5ca96c8a9a74413bff544306af10586"
I20240501 20:12:41.299216 14560 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:41.299094 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "3c0f925474374c18a6cd80cff94d4295"
I20240501 20:12:41.299373 14694 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:41.302237 14560 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 2.
I20240501 20:12:41.302441 14694 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 2.
I20240501 20:12:41.302685 14234 leader_election.cc:302] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 8 voters: 5 yes votes; 0 no votes. yes voters: 0e898d8ee23441908d1593c561b368ad, 692bb14bda8046b9827f1091b0946088, a5ca96c8a9a74413bff544306af10586, c12d6cddb7f242b6af650a10bbc8eb69, c3224a319e2f4cb8a84bf9528209a9fb; no voters:
I20240501 20:12:41.303110 18408 raft_consensus.cc:2781] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:12:41.303289 18408 raft_consensus.cc:686] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 2 LEADER]: Becoming Leader. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Running, Role: LEADER
I20240501 20:12:41.303930 18408 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:41.312758 13765 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 reported cstate change: term changed from 1 to 2, leader changed from c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) to 692bb14bda8046b9827f1091b0946088 (127.28.136.68). New cstate: current_term: 2 leader_uuid: "692bb14bda8046b9827f1091b0946088" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:41.347277 14296 leader_election.cc:288] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [CANDIDATE]: Term 2 election: Requested vote from peers c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939), c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849), 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579), a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397), 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621), aeb70769b705493bb5196527311b142b (127.28.136.72:39739), 1899220968384feebca3562bd7741a21 (127.28.136.74:45465)
I20240501 20:12:41.365262 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "1899220968384feebca3562bd7741a21"
I20240501 20:12:41.365653 16432 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:41.368321 16432 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 2.
I20240501 20:12:41.369813 16302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1592bebd6b4a858987847446f15a16" candidate_uuid: "692bb14bda8046b9827f1091b0946088" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "aeb70769b705493bb5196527311b142b"
I20240501 20:12:41.370177 16302 raft_consensus.cc:3037] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:12:41.372661 16302 raft_consensus.cc:2454] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 692bb14bda8046b9827f1091b0946088 in term 2.
I20240501 20:12:41.380137 14038 consensus_queue.cc:237] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
W20240501 20:12:41.383998 13973 consensus_peers.cc:485] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60c32a552af1499dab8884bae2287ce3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:41.385470 16302 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.385646 14694 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.386575 18368 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.386974 14560 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.387543 18083 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.388124 13905 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.388803 18083 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.390851 18083 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.390905 14430 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.391098 14169 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:41.392814 18368 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.393280 18368 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:41.406708 18082 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:41.411455 14560 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:41.411518 13902 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:41.414168 16302 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:41.417930 13761 catalog_manager.cc:5484] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:41.431414 14429 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:41.434535 14694 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:41.420655 14169 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:41.695731 18535 ts_tablet_manager.cc:902] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:41.697739 18535 tablet_copy_client.cc:308] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:41.698415 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet 77ef10335c1642818694b42ab006a66f from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:45703)
I20240501 20:12:41.698592 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet 77ef10335c1642818694b42ab006a66f from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:45703: session id = 2a31af84f57c4ef18f0d98ff044229b6-77ef10335c1642818694b42ab006a66f
I20240501 20:12:41.700702 14714 tablet_copy_source_session.cc:205] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:41.701870 18535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ef10335c1642818694b42ab006a66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:41.706427 18535 tablet_copy_client.cc:783] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:41.706651 18535 tablet_copy_client.cc:647] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:41.708252 18535 tablet_copy_client.cc:523] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:41.710686 18535 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:41.712134 14560 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:12:41.712774 18408 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.744109 16432 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:12:41.744949 18408 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.759097 18535 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:41.759634 18535 tablet_bootstrap.cc:492] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:41.759979 18535 ts_tablet_manager.cc:1367] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.049s user 0.036s sys 0.009s
I20240501 20:12:41.760871 18535 raft_consensus.cc:348] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:41.761402 18535 raft_consensus.cc:729] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:41.761667 18535 consensus_queue.cc:260] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:41.766287 18535 ts_tablet_manager.cc:1398] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20240501 20:12:41.766973 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-77ef10335c1642818694b42ab006a66f received from {username='slave'} at 127.28.136.73:45703
I20240501 20:12:41.767115 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-77ef10335c1642818694b42ab006a66f on tablet 77ef10335c1642818694b42ab006a66f with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:41.778409 14038 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:12:41.779906 18408 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.831485 14169 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:12:41.832195 18408 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.840229 13905 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:12:41.840829 14694 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:12:41.843154 16303 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:12:41.852167 18559 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20240501 20:12:41.861435 18408 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:41.863498 18566 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240501 20:12:41.893071 18571 ts_tablet_manager.cc:902] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:41.900527 18571 tablet_copy_client.cc:308] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:41.901306 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet 60c32a552af1499dab8884bae2287ce3 from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:35733)
I20240501 20:12:41.901480 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet 60c32a552af1499dab8884bae2287ce3 from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:35733: session id = 1899220968384feebca3562bd7741a21-60c32a552af1499dab8884bae2287ce3
I20240501 20:12:41.903472 14058 tablet_copy_source_session.cc:205] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:41.904781 18571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c32a552af1499dab8884bae2287ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:41.909885 18571 tablet_copy_client.cc:783] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:41.910215 18571 tablet_copy_client.cc:647] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:41.911825 18571 tablet_copy_client.cc:523] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:41.914453 18571 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:41.950029 18571 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:41.950513 18571 tablet_bootstrap.cc:492] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:41.950798 18571 ts_tablet_manager.cc:1367] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.036s user 0.032s sys 0.004s
I20240501 20:12:41.951455 18571 raft_consensus.cc:348] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:41.951706 18571 raft_consensus.cc:729] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:41.951881 18571 consensus_queue.cc:260] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:41.952383 18571 ts_tablet_manager.cc:1398] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:12:41.953239 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session 1899220968384feebca3562bd7741a21-60c32a552af1499dab8884bae2287ce3 received from {username='slave'} at 127.28.136.74:35733
I20240501 20:12:41.953388 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session 1899220968384feebca3562bd7741a21-60c32a552af1499dab8884bae2287ce3 on tablet 60c32a552af1499dab8884bae2287ce3 with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:42.232525 16567 raft_consensus.cc:1203] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:42.288950 16432 raft_consensus.cc:1203] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:42.371381 14296 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:42.374240 16303 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:12:42.374364 14694 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:12:42.374933 16432 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:12:42.375010 18565 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:42.375625 13905 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:12:42.375634 18565 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:42.375901 14563 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:12:42.376189 18565 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:42.376677 18565 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:42.377118 18567 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:42.378636 14169 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:12:42.379465 18408 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:42.383095 18408 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.387406 14563 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.387570 14169 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.391225 16432 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.392652 16303 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.395447 13765 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 9 to 11, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New cstate: current_term: 2 leader_uuid: "692bb14bda8046b9827f1091b0946088" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.400718 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe1592bebd6b4a858987847446f15a16 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:42.400415 14693 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.402848 13905 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.418601 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet fe1592bebd6b4a858987847446f15a16 with delete_type TABLET_DATA_TOMBSTONED (TS c3224a319e2f4cb8a84bf9528209a9fb not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:56540
I20240501 20:12:42.419540 18614 tablet_replica.cc:330] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: stopping tablet replica
I20240501 20:12:42.419757 18614 raft_consensus.cc:2229] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:42.419974 18614 raft_consensus.cc:2258] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:42.420864 18614 ts_tablet_manager.cc:1875] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:42.426453 18614 ts_tablet_manager.cc:1888] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240501 20:12:42.426576 18614 log.cc:1196] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/fe1592bebd6b4a858987847446f15a16
I20240501 20:12:42.427342 13749 catalog_manager.cc:4830] TS c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): tablet fe1592bebd6b4a858987847446f15a16 (table rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]) successfully deleted
I20240501 20:12:42.854941 18319 raft_consensus.cc:1050] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:42.861766 18319 consensus_queue.cc:237] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:42.865526 16567 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.866194 18324 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.866405 16303 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.867295 18652 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.869740 14563 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.870615 14036 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.870874 14169 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.871446 18324 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.872126 18324 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.872514 14429 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.872658 18324 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240501 20:12:42.879145 18652 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.897365 13905 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.896543 18652 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:42.898236 18319 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.899032 14562 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:42.900640 14169 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:42.902807 16567 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:42.904073 14038 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:42.908123 14429 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:42.908803 13757 catalog_manager.cc:5484] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:42.912259 16303 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:42.925529 18371 raft_consensus.cc:1050] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:42.926949 18371 consensus_queue.cc:237] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:42.935622 13905 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:42.939496 16432 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.940156 18607 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.944357 16303 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.945597 18607 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.956339 14429 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.957042 14166 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.958563 18371 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.958783 13905 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.958899 18607 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.959863 18607 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.960892 14694 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.960919 14562 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:42.962199 18607 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.962742 18607 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:42.979729 18671 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.984509 16432 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.985194 16303 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.988116 13905 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:42.991662 13765 catalog_manager.cc:5484] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b reported cstate change: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.000490 14166 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.001560 14562 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.010186 14693 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.016024 14429 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.017853 14036 consensus_queue.cc:237] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:43.023053 16432 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.023921 14562 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.024096 14693 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.024526 18667 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.025069 18671 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.025882 16303 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.027972 18371 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.028596 13905 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.029145 18674 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.029901 18674 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.030139 14429 raft_consensus.cc:1261] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.030755 18674 consensus_queue.cc:1035] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.034929 18674 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.035881 16302 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.042388 13749 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 60c32a552af1499dab8884bae2287ce3 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:43.042799 13905 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.044766 14693 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.050688 13764 catalog_manager.cc:5484] T 60c32a552af1499dab8884bae2287ce3 P aeb70769b705493bb5196527311b142b reported cstate change: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.057806 14430 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.060696 14147 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c32a552af1499dab8884bae2287ce3 with delete_type TABLET_DATA_TOMBSTONED (TS 0e898d8ee23441908d1593c561b368ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44418
I20240501 20:12:43.064934 14562 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.064705 16432 raft_consensus.cc:2932] T 60c32a552af1499dab8884bae2287ce3 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:43.068751 18687 tablet_replica.cc:330] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: stopping tablet replica
I20240501 20:12:43.069097 18687 raft_consensus.cc:2229] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:43.069481 18687 raft_consensus.cc:2258] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:43.070847 18687 ts_tablet_manager.cc:1875] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:43.079196 18687 ts_tablet_manager.cc:1888] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:43.079432 18687 log.cc:1196] T 60c32a552af1499dab8884bae2287ce3 P 0e898d8ee23441908d1593c561b368ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/60c32a552af1499dab8884bae2287ce3
I20240501 20:12:43.080199 13751 catalog_manager.cc:4830] TS 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): tablet 60c32a552af1499dab8884bae2287ce3 (table rebalance_test_table_0 [id=31066ee991a64c6db75417ba2b661b14]) successfully deleted
I20240501 20:12:43.527904 14693 consensus_queue.cc:237] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:43.531023 14562 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.531003 14430 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.531412 14166 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.531503 16302 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.532047 18654 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.532644 16567 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.532521 18710 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.533015 13905 raft_consensus.cc:1261] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:43.533136 18705 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.533725 18319 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.534734 18654 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.535817 18705 consensus_queue.cc:1035] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:43.536788 18705 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:43.538908 14562 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:43.540069 13905 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:43.543108 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 77ef10335c1642818694b42ab006a66f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:43.542793 14430 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:43.543002 14166 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:43.543916 13765 catalog_manager.cc:5484] T 77ef10335c1642818694b42ab006a66f P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:43.544804 16302 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:43.551856 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ef10335c1642818694b42ab006a66f with delete_type TABLET_DATA_TOMBSTONED (TS c3224a319e2f4cb8a84bf9528209a9fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56540
I20240501 20:12:43.552599 18712 tablet_replica.cc:330] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: stopping tablet replica
I20240501 20:12:43.552587 16567 raft_consensus.cc:2932] T 77ef10335c1642818694b42ab006a66f P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:43.554145 18712 raft_consensus.cc:2229] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:43.554543 18712 raft_consensus.cc:2258] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:43.557545 18712 ts_tablet_manager.cc:1875] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:43.572122 18712 ts_tablet_manager.cc:1888] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:43.572470 18712 log.cc:1196] T 77ef10335c1642818694b42ab006a66f P c3224a319e2f4cb8a84bf9528209a9fb: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/77ef10335c1642818694b42ab006a66f
I20240501 20:12:43.573307 13749 catalog_manager.cc:4830] TS c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): tablet 77ef10335c1642818694b42ab006a66f (table rebalance_test_table_5 [id=fd7b9986cacc49d281b343bd7b74dc9f]) successfully deleted
I20240501 20:12:43.714253 16416 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:43.716449 16549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:43.723742 14147 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:43.838732 16284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:43.840169 14674 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:43.848012 14279 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:43.856551 14542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:43.873186 13884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:43.897791 14410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:43.912614 14017 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:44.029492 14560 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:44.033130 14169 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.033787 16567 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.033787 13905 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.034153 14038 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.034368 14693 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.034787 18751 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.035180 14429 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.035394 18751 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.036139 18751 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:12:44.036432 14497 consensus_peers.cc:485] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eeaff69149ee4707beba512799b0d0df. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:44.036514 18768 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.037253 18751 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.037812 18686 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.054606 18768 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:44.056360 14169 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:44.058408 14036 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:44.059940 13905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:44.066418 13762 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:44.067148 14693 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:44.089330 16567 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:44.093179 14430 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:44.112658 13905 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:44.116619 16432 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.116639 14169 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.118603 18090 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.119040 18787 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.121387 14038 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.121500 14693 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.122470 14296 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.122817 14430 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:44.123481 18090 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.123857 18788 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.124505 18788 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.124881 18787 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:44.142119 18090 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:44.146395 14296 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:44.148763 14169 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
W20240501 20:12:44.149504 13842 consensus_peers.cc:485] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 -> Peer 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): Couldn't send request to peer 2a31af84f57c4ef18f0d98ff044229b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1840c628fc114b309da99d3c8a91656a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:44.147576 14693 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:44.154827 13765 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:44.159969 14038 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:44.161201 14430 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:44.163838 16432 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } } }
I20240501 20:12:44.572785 18820 ts_tablet_manager.cc:902] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:44.574335 18820 tablet_copy_client.cc:308] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:44.575210 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet eeaff69149ee4707beba512799b0d0df from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:34649)
I20240501 20:12:44.575398 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet eeaff69149ee4707beba512799b0d0df from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:34649: session id = 1899220968384feebca3562bd7741a21-eeaff69149ee4707beba512799b0d0df
I20240501 20:12:44.577646 14583 tablet_copy_source_session.cc:205] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:44.579366 18820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:44.585213 18820 tablet_copy_client.cc:783] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:44.585492 18820 tablet_copy_client.cc:647] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:44.587158 18820 tablet_copy_client.cc:523] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:44.590121 18820 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:44.602962 18826 ts_tablet_manager.cc:902] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: Initiating tablet copy from peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939)
I20240501 20:12:44.604688 18826 tablet_copy_client.cc:308] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:42939
I20240501 20:12:44.611941 13926 tablet_copy_service.cc:140] P c12d6cddb7f242b6af650a10bbc8eb69: Received BeginTabletCopySession request for tablet 1840c628fc114b309da99d3c8a91656a from peer 2a31af84f57c4ef18f0d98ff044229b6 ({username='slave'} at 127.28.136.73:47083)
I20240501 20:12:44.612130 13926 tablet_copy_service.cc:161] P c12d6cddb7f242b6af650a10bbc8eb69: Beginning new tablet copy session on tablet 1840c628fc114b309da99d3c8a91656a from peer 2a31af84f57c4ef18f0d98ff044229b6 at {username='slave'} at 127.28.136.73:47083: session id = 2a31af84f57c4ef18f0d98ff044229b6-1840c628fc114b309da99d3c8a91656a
I20240501 20:12:44.614193 13926 tablet_copy_source_session.cc:205] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:44.615675 18826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:44.621508 18826 tablet_copy_client.cc:783] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:44.621774 18826 tablet_copy_client.cc:647] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:44.623564 18826 tablet_copy_client.cc:523] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:44.626778 18826 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap starting.
I20240501 20:12:44.631002 18820 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:44.631451 18820 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:44.631762 18820 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.042s user 0.034s sys 0.005s
I20240501 20:12:44.632607 18820 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:44.632961 18820 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:44.633180 18820 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:44.633972 18820 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240501 20:12:44.634864 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session 1899220968384feebca3562bd7741a21-eeaff69149ee4707beba512799b0d0df received from {username='slave'} at 127.28.136.74:34649
I20240501 20:12:44.635023 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session 1899220968384feebca3562bd7741a21-eeaff69149ee4707beba512799b0d0df on tablet eeaff69149ee4707beba512799b0d0df with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:44.673048 18826 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:44.673516 18826 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: Bootstrap complete.
I20240501 20:12:44.673843 18826 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: Time spent bootstrapping tablet: real 0.047s user 0.033s sys 0.007s
I20240501 20:12:44.674459 18826 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:44.674721 18826 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a31af84f57c4ef18f0d98ff044229b6, State: Initialized, Role: LEARNER
I20240501 20:12:44.674963 18826 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: NON_VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: true } }
I20240501 20:12:44.675762 18826 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:44.676581 13926 tablet_copy_service.cc:342] P c12d6cddb7f242b6af650a10bbc8eb69: Request end of tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-1840c628fc114b309da99d3c8a91656a received from {username='slave'} at 127.28.136.73:47083
I20240501 20:12:44.676757 13926 tablet_copy_service.cc:433] P c12d6cddb7f242b6af650a10bbc8eb69: ending tablet copy session 2a31af84f57c4ef18f0d98ff044229b6-1840c628fc114b309da99d3c8a91656a on tablet 1840c628fc114b309da99d3c8a91656a with peer 2a31af84f57c4ef18f0d98ff044229b6
I20240501 20:12:44.962332 16432 raft_consensus.cc:1203] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:45.014154 16567 raft_consensus.cc:1203] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:45.396541 18857 raft_consensus.cc:1050] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: attempting to promote NON_VOTER 2a31af84f57c4ef18f0d98ff044229b6 to VOTER
I20240501 20:12:45.397675 18857 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:45.401082 16567 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 LEARNER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.402369 18801 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.404330 16432 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.406261 18801 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.407491 14169 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.408193 18801 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.428961 14694 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.429028 14038 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.429165 14296 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.429659 18801 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.430080 18844 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.430862 18844 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.431998 14430 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.432881 18879 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.442688 18844 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.444130 14430 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.448908 14693 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.453372 16568 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.455879 14169 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.457105 16433 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.459529 14036 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.461117 13765 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.470633 14296 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.497107 18892 raft_consensus.cc:1050] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:45.498778 18892 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:45.510468 14169 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.511113 16433 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.511909 18856 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.513541 18856 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.522972 13905 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } }
I20240501 20:12:45.525744 13902 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.526343 16568 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.526535 18893 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.526790 14693 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.528216 14169 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.528235 16433 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.528683 18893 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.528856 18883 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.529701 18883 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.533406 16568 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.533634 14036 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.534207 18893 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.534224 18857 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.535845 14430 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.536967 14694 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.538416 18801 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.539044 18801 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.539851 14430 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:45.540568 18893 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.545892 18857 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.549885 14169 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.551565 14430 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.556393 16568 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.558128 18856 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:45.558463 14296 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.559089 18844 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.561854 14296 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.564096 13764 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
W20240501 20:12:45.565229 18894 log.cc:924] Time spent T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240501 20:12:45.570089 13752 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1840c628fc114b309da99d3c8a91656a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:45.573156 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet 1840c628fc114b309da99d3c8a91656a with delete_type TABLET_DATA_TOMBSTONED (TS c3224a319e2f4cb8a84bf9528209a9fb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56540
I20240501 20:12:45.574146 18856 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.577504 14036 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.579188 14694 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.580529 16433 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } }
I20240501 20:12:45.593407 14169 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.594033 16432 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.597611 14693 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.598250 13762 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.611033 14429 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.629323 18904 tablet_replica.cc:330] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: stopping tablet replica
I20240501 20:12:45.628700 13902 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.629637 18904 raft_consensus.cc:2229] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:45.630462 18904 raft_consensus.cc:2258] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:45.631764 18904 ts_tablet_manager.cc:1875] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:45.635593 16567 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.639532 14562 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:45.643088 16432 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.643206 14693 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.644009 14038 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.643923 18906 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.644228 14429 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.644309 13905 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.644521 18906 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.645005 18906 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.646152 18856 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.647183 18892 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.647606 16567 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:45.648451 18892 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:45.649593 18906 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.652067 14429 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.652659 18904 ts_tablet_manager.cc:1888] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:45.652909 18904 log.cc:1196] T 1840c628fc114b309da99d3c8a91656a P c3224a319e2f4cb8a84bf9528209a9fb: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-1/wal/wals/1840c628fc114b309da99d3c8a91656a
I20240501 20:12:45.653019 16567 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.653308 14038 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.654698 13749 catalog_manager.cc:4830] TS c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): tablet 1840c628fc114b309da99d3c8a91656a (table rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]) successfully deleted
I20240501 20:12:45.658540 13762 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.660135 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eeaff69149ee4707beba512799b0d0df with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:45.661247 16432 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.662060 14693 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.662205 13905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:45.667507 14147 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaff69149ee4707beba512799b0d0df with delete_type TABLET_DATA_TOMBSTONED (TS 0e898d8ee23441908d1593c561b368ad not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:44418
I20240501 20:12:45.680114 18912 tablet_replica.cc:330] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: stopping tablet replica
I20240501 20:12:45.680363 18912 raft_consensus.cc:2229] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:45.680701 18912 raft_consensus.cc:2258] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:45.683297 18912 ts_tablet_manager.cc:1875] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:45.689759 18912 ts_tablet_manager.cc:1888] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 20:12:45.689975 18912 log.cc:1196] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/eeaff69149ee4707beba512799b0d0df
I20240501 20:12:45.691455 13751 catalog_manager.cc:4830] TS 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): tablet eeaff69149ee4707beba512799b0d0df (table rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]) successfully deleted
I20240501 20:12:45.846040 16549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:45.860301 14147 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:45.861809 16416 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:45.906512 14542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:45.947476 13884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:45.949867 14279 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:45.963444 14410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:45.966992 16284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:45.978679 14674 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:46.039335 14017 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:46.341869 16549 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:46.357209 14542 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:46.362833 14674 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:12:46.363293 14410 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:46.367352 16416 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:46.390688 14279 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.398679 16284 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:12:46.404006 13884 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:12:46.467062 14017 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:12:46.477756 14147 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:12:46.653651 14429 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
I20240501 20:12:46.654997 14298 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } }
W20240501 20:12:46.658277 14365 consensus_peers.cc:485] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Couldn't send request to peer 692bb14bda8046b9827f1091b0946088. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:46.658279 16432 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.659065 16567 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.659436 13902 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.659512 14038 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.659813 14562 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.659926 19041 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.660535 19041 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.662935 14693 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.661100 19041 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20240501 20:12:46.670516 19045 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240501 20:12:46.671303 18855 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.672024 18855 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.679630 14693 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:46.683043 18855 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
W20240501 20:12:46.687448 14629 consensus_peers.cc:485] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af0f6a4c2e9c43509502a242b596d9da. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:46.689931 16303 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.690136 14429 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.690546 16567 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.690799 18705 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.690438 13902 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:46.691658 14562 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.691843 18705 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.692381 18705 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.693588 18958 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.697535 16433 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 20:12:46.698534 14233 consensus_peers.cc:485] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 -> Peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): Couldn't send request to peer c3224a319e2f4cb8a84bf9528209a9fb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:46.699224 14694 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:46.703622 18567 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:46.704272 14560 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:46.704639 16568 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:46.706923 14038 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:46.710450 13764 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:46.712512 14560 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:12:46.713542 18567 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:46.715229 14692 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:12:46.715938 18567 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:46.716626 18710 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.718123 14429 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.732419 16303 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:12:46.737243 18567 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:46.744349 14166 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:12:46.744627 13902 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:12:46.745160 18567 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:46.747558 18567 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:46.753372 13757 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.754348 18567 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:46.757990 14166 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:46.758244 14563 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:46.765168 13757 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 11 to 12, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New cstate: current_term: 2 leader_uuid: "692bb14bda8046b9827f1091b0946088" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:46.768631 13902 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:46.773310 16433 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:46.779775 14694 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:46.793319 16302 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:46.827126 14038 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:46.830049 14036 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.831269 18958 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.833683 13902 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.834686 18705 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.864395 13902 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.863830 16567 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.865038 18671 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:12:46.868450 13973 consensus_peers.cc:485] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb -> Peer 1899220968384feebca3562bd7741a21 (127.28.136.74:45465): Couldn't send request to peer 1899220968384feebca3562bd7741a21. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8892d6dce05140babdeeb6c90c366b3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:46.873781 14563 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.875361 16303 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.875798 14693 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.875761 16302 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.876331 18761 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.878094 18761 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.880417 16432 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:46.888540 16568 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.889276 18761 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.891196 14298 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.891773 18761 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.902729 14562 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:46.903376 18761 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:46.911669 14036 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.915792 13905 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.928313 18671 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.931105 14298 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.945510 14694 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.946153 14563 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:46.951658 13757 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:46.998261 14694 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } }
I20240501 20:12:46.998071 16567 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:47.002903 16432 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.003012 16568 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.003692 18710 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.003340 16300 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:47.004212 19052 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.007277 14429 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.008208 18710 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.008291 13902 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } } }
I20240501 20:12:47.026736 14298 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.028221 18710 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:12:47.040118 14627 consensus_peers.cc:485] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 -> Peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): Couldn't send request to peer c3224a319e2f4cb8a84bf9528209a9fb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.048671 18710 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:47.050592 16433 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:47.061517 14166 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.062347 18705 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.060322 16567 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:47.065901 13902 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.066568 18705 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.070521 19112 ts_tablet_manager.cc:902] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Initiating tablet copy from peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:47.072875 14429 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:47.075027 13759 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 10 to 11, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:47.082031 14298 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:47.092501 14166 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:47.092805 13905 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } }
I20240501 20:12:47.105641 14694 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:47.109351 14169 consensus_queue.cc:237] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
W20240501 20:12:47.110781 14628 consensus_peers.cc:485] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 -> Peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Couldn't send request to peer a5ca96c8a9a74413bff544306af10586. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.112188 16567 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 20:12:47.112571 14627 consensus_peers.cc:485] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 -> Peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849): Couldn't send request to peer c3224a319e2f4cb8a84bf9528209a9fb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.113395 16432 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.116113 18705 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.116397 18958 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.116647 14429 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.117569 18958 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.123577 14298 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.123860 14166 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.124233 18705 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.125682 18958 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.129132 16432 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.128804 14298 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.129746 19023 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.129981 18710 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.134519 16567 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.137372 13757 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 11 to 12, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.138121 14166 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.145602 14429 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.144436 16433 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.166574 13904 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.167825 14036 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.167773 19023 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.168663 19023 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.178699 14429 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.180050 19023 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:12:47.189896 14105 consensus_peers.cc:485] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3808b45349de4959b929a99735e5db53. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.195408 13902 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.199359 19105 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.201601 13902 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.208185 19126 ts_tablet_manager.cc:902] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Initiating tablet copy from peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031)
I20240501 20:12:47.214071 14694 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.215946 19023 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.247483 19023 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.262504 13902 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.271817 19126 tablet_copy_client.cc:308] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.68:38031
I20240501 20:12:47.274736 14320 tablet_copy_service.cc:140] P 692bb14bda8046b9827f1091b0946088: Received BeginTabletCopySession request for tablet fe1592bebd6b4a858987847446f15a16 from peer c3224a319e2f4cb8a84bf9528209a9fb ({username='slave'} at 127.28.136.66:34685)
I20240501 20:12:47.275019 14320 tablet_copy_service.cc:161] P 692bb14bda8046b9827f1091b0946088: Beginning new tablet copy session on tablet fe1592bebd6b4a858987847446f15a16 from peer c3224a319e2f4cb8a84bf9528209a9fb at {username='slave'} at 127.28.136.66:34685: session id = c3224a319e2f4cb8a84bf9528209a9fb-fe1592bebd6b4a858987847446f15a16
I20240501 20:12:47.279250 14038 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.280865 14320 tablet_copy_source_session.cc:205] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:47.286427 19126 ts_tablet_manager.cc:1875] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:47.289331 19141 ts_tablet_manager.cc:902] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:47.291141 13762 catalog_manager.cc:5484] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.299702 14298 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.304340 14694 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.310698 19139 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.318918 14298 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.324657 16435 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.329051 19148 ts_tablet_manager.cc:902] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:47.332430 14429 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.357580 19141 tablet_copy_client.cc:308] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:47.358856 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet af0f6a4c2e9c43509502a242b596d9da from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:35471)
I20240501 20:12:47.359069 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet af0f6a4c2e9c43509502a242b596d9da from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:35471: session id = 1899220968384feebca3562bd7741a21-af0f6a4c2e9c43509502a242b596d9da
I20240501 20:12:47.359318 19148 tablet_copy_client.cc:308] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.66:44849
I20240501 20:12:47.361580 19126 ts_tablet_manager.cc:1888] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.10
I20240501 20:12:47.361941 14714 tablet_copy_source_session.cc:205] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:47.363437 19141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af0f6a4c2e9c43509502a242b596d9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:47.362715 14058 tablet_copy_service.cc:140] P c3224a319e2f4cb8a84bf9528209a9fb: Received BeginTabletCopySession request for tablet 8892d6dce05140babdeeb6c90c366b3b from peer 1899220968384feebca3562bd7741a21 ({username='slave'} at 127.28.136.74:35733)
I20240501 20:12:47.368284 14058 tablet_copy_service.cc:161] P c3224a319e2f4cb8a84bf9528209a9fb: Beginning new tablet copy session on tablet 8892d6dce05140babdeeb6c90c366b3b from peer 1899220968384feebca3562bd7741a21 at {username='slave'} at 127.28.136.74:35733: session id = 1899220968384feebca3562bd7741a21-8892d6dce05140babdeeb6c90c366b3b
I20240501 20:12:47.368664 19126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1592bebd6b4a858987847446f15a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:47.371407 14058 tablet_copy_source_session.cc:205] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:47.372984 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8892d6dce05140babdeeb6c90c366b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:47.380497 19141 tablet_copy_client.cc:783] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:47.380793 19141 tablet_copy_client.cc:647] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:47.381494 19148 tablet_copy_client.cc:783] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:47.381712 19148 tablet_copy_client.cc:647] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:47.384372 19141 tablet_copy_client.cc:523] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:47.385077 19148 tablet_copy_client.cc:523] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:47.390811 14169 consensus_queue.cc:237] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:47.395406 14563 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
I20240501 20:12:47.397363 19112 tablet_copy_client.cc:308] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:32991
I20240501 20:12:47.397792 19148 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:47.398588 14451 tablet_copy_service.cc:140] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Received BeginTabletCopySession request for tablet c6213df9f24a4e63aec805f7dfacc75d from peer 692bb14bda8046b9827f1091b0946088 ({username='slave'} at 127.28.136.68:42323)
I20240501 20:12:47.398772 14451 tablet_copy_service.cc:161] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Beginning new tablet copy session on tablet c6213df9f24a4e63aec805f7dfacc75d from peer 692bb14bda8046b9827f1091b0946088 at {username='slave'} at 127.28.136.68:42323: session id = 692bb14bda8046b9827f1091b0946088-c6213df9f24a4e63aec805f7dfacc75d
I20240501 20:12:47.401695 19141 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: Bootstrap starting.
I20240501 20:12:47.401918 14451 tablet_copy_source_session.cc:205] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:47.403928 19112 ts_tablet_manager.cc:1875] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:47.411454 19126 tablet_copy_client.cc:783] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:47.411871 19126 tablet_copy_client.cc:647] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:47.413349 16433 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.414064 19156 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.418605 14429 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } }
W20240501 20:12:47.422780 14365 consensus_peers.cc:485] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Couldn't send request to peer 692bb14bda8046b9827f1091b0946088. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:47.422785 14366 consensus_peers.cc:485] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 -> Peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Couldn't send request to peer 0e898d8ee23441908d1593c561b368ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.424257 16567 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 20:12:47.424481 14496 consensus_peers.cc:485] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 -> Peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Couldn't send request to peer 692bb14bda8046b9827f1091b0946088. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.425395 13904 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.425746 16435 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.426395 14694 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.426326 19042 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.426852 19042 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.427213 19045 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.427750 19045 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.427840 14562 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.428354 19045 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.429510 16300 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.429932 14693 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.433447 18905 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.434165 18905 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.435297 14038 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:47.436964 19051 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:47.438604 19051 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.444955 14694 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.448172 14562 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.450646 16435 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.453002 14038 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.457147 13757 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.469166 16567 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.480242 19112 ts_tablet_manager.cc:1888] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:12:47.480659 19112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:47.488004 13904 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.496320 14038 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.497370 19156 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.497416 14429 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.498971 19126 tablet_copy_client.cc:523] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:47.498962 19156 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.500470 19112 tablet_copy_client.cc:783] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: tablet copy: Starting download of 0 data blocks...
W20240501 20:12:47.500553 14105 consensus_peers.cc:485] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad -> Peer aeb70769b705493bb5196527311b142b (127.28.136.72:39739): Couldn't send request to peer aeb70769b705493bb5196527311b142b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39eb086c2b80482a98ff2baae96b9f60. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.500908 19112 tablet_copy_client.cc:647] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:47.503458 13904 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.504537 14694 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.505306 19130 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.505923 14036 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.505985 19130 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.507591 13902 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.508324 18905 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.508560 19126 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:47.510816 18905 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:47.513319 19137 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.514766 14694 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:47.522145 13757 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:47.538594 14429 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.539392 19137 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.540372 19130 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.542399 14036 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.547670 13904 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.547775 14693 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.558218 16435 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.558526 13759 catalog_manager.cc:5484] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad reported cstate change: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:47.559093 19130 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.584044 14563 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:47.595997 19112 tablet_copy_client.cc:523] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:47.629572 19148 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:47.630115 19148 tablet_bootstrap.cc:492] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: Bootstrap complete.
W20240501 20:12:47.633395 14497 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.69:32991, user_credentials={real_user=slave}} blocked reactor thread for 109627us
I20240501 20:12:47.633595 13904 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:47.635542 19196 ts_tablet_manager.cc:902] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:47.637329 19195 ts_tablet_manager.cc:902] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:47.637713 19137 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:47.638929 16302 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:47.630447 16435 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.647485 19141 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:47.647958 19141 tablet_bootstrap.cc:492] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: Bootstrap complete.
I20240501 20:12:47.644047 16433 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:47.650599 19112 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:47.650411 14038 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:47.653323 19141 ts_tablet_manager.cc:1367] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.252s user 0.049s sys 0.012s
I20240501 20:12:47.654193 19141 raft_consensus.cc:348] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:47.654541 19141 raft_consensus.cc:729] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:47.658685 14429 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:47.656944 19141 consensus_queue.cc:260] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:47.664631 19141 ts_tablet_manager.cc:1398] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 20:12:47.665465 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session 1899220968384feebca3562bd7741a21-af0f6a4c2e9c43509502a242b596d9da received from {username='slave'} at 127.28.136.74:35471
I20240501 20:12:47.665607 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session 1899220968384feebca3562bd7741a21-af0f6a4c2e9c43509502a242b596d9da on tablet af0f6a4c2e9c43509502a242b596d9da with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:47.630470 19148 ts_tablet_manager.cc:1367] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: Time spent bootstrapping tablet: real 0.233s user 0.051s sys 0.008s
I20240501 20:12:47.678244 14430 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.687330 19206 ts_tablet_manager.cc:902] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Initiating tablet copy from peer 3c0f925474374c18a6cd80cff94d4295 (127.28.136.71:43621)
I20240501 20:12:47.689327 19148 raft_consensus.cc:348] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:47.689757 19148 raft_consensus.cc:729] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1899220968384feebca3562bd7741a21, State: Initialized, Role: LEARNER
I20240501 20:12:47.690022 19148 consensus_queue.cc:260] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: NON_VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: true } }
I20240501 20:12:47.704066 19148 ts_tablet_manager.cc:1398] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240501 20:12:47.705229 14562 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } } }
I20240501 20:12:47.707798 14058 tablet_copy_service.cc:342] P c3224a319e2f4cb8a84bf9528209a9fb: Request end of tablet copy session 1899220968384feebca3562bd7741a21-8892d6dce05140babdeeb6c90c366b3b received from {username='slave'} at 127.28.136.74:35733
I20240501 20:12:47.707955 14058 tablet_copy_service.cc:433] P c3224a319e2f4cb8a84bf9528209a9fb: ending tablet copy session 1899220968384feebca3562bd7741a21-8892d6dce05140babdeeb6c90c366b3b on tablet 8892d6dce05140babdeeb6c90c366b3b with peer 1899220968384feebca3562bd7741a21
I20240501 20:12:47.714237 19195 tablet_copy_client.cc:308] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38579
I20240501 20:12:47.715041 14189 tablet_copy_service.cc:140] P 0e898d8ee23441908d1593c561b368ad: Received BeginTabletCopySession request for tablet 3808b45349de4959b929a99735e5db53 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:35535)
I20240501 20:12:47.715269 14189 tablet_copy_service.cc:161] P 0e898d8ee23441908d1593c561b368ad: Beginning new tablet copy session on tablet 3808b45349de4959b929a99735e5db53 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:35535: session id = aeb70769b705493bb5196527311b142b-3808b45349de4959b929a99735e5db53
I20240501 20:12:47.718099 14189 tablet_copy_source_session.cc:205] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:47.721333 14563 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } }
I20240501 20:12:47.729091 19195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3808b45349de4959b929a99735e5db53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:47.739267 19196 tablet_copy_client.cc:308] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:47.743201 13904 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.743356 14694 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.744114 14038 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.744189 16567 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.745013 19156 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.745776 16435 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.745744 19185 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:12:47.746357 14497 consensus_peers.cc:485] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 -> Peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): Couldn't send request to peer 0e898d8ee23441908d1593c561b368ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.746919 19156 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.747467 18905 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.748618 19206 tablet_copy_client.cc:308] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.71:43621
I20240501 20:12:47.749500 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 from peer a5ca96c8a9a74413bff544306af10586 ({username='slave'} at 127.28.136.70:40385)
I20240501 20:12:47.749675 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 from peer a5ca96c8a9a74413bff544306af10586 at {username='slave'} at 127.28.136.70:40385: session id = a5ca96c8a9a74413bff544306af10586-4e21fb0e6bac46fbbd29bbd2f1bf43b4
I20240501 20:12:47.750556 18905 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.752616 14714 tablet_copy_source_session.cc:205] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:47.758245 19195 tablet_copy_client.cc:783] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:47.758584 19195 tablet_copy_client.cc:647] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:47.760860 19195 tablet_copy_client.cc:523] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:47.769501 19195 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:47.774031 13902 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:47.777608 18905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.778363 13905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.787060 13759 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 10 to 11, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.793370 19206 ts_tablet_manager.cc:1875] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:47.800886 19126 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:47.802999 19126 tablet_bootstrap.cc:492] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap complete.
I20240501 20:12:47.811031 19126 ts_tablet_manager.cc:1367] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.296s user 0.065s sys 0.006s
I20240501 20:12:47.812115 19126 raft_consensus.cc:348] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } }
I20240501 20:12:47.812498 19126 raft_consensus.cc:729] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: LEARNER
I20240501 20:12:47.812299 16567 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.812772 19126 consensus_queue.cc:260] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } }
I20240501 20:12:47.818238 19126 ts_tablet_manager.cc:1398] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20240501 20:12:47.819023 13840 consensus_peers.cc:485] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 -> Peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397): Couldn't send request to peer a5ca96c8a9a74413bff544306af10586. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:47.820276 16433 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.821000 14320 tablet_copy_service.cc:342] P 692bb14bda8046b9827f1091b0946088: Request end of tablet copy session c3224a319e2f4cb8a84bf9528209a9fb-fe1592bebd6b4a858987847446f15a16 received from {username='slave'} at 127.28.136.66:34685
I20240501 20:12:47.821002 18983 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.821219 14320 tablet_copy_service.cc:433] P 692bb14bda8046b9827f1091b0946088: ending tablet copy session c3224a319e2f4cb8a84bf9528209a9fb-fe1592bebd6b4a858987847446f15a16 on tablet fe1592bebd6b4a858987847446f15a16 with peer c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:47.821451 14430 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.826874 19185 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.831916 14038 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.842185 16568 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.842476 14298 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.844640 14169 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.844651 14694 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.845510 18983 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.846287 18983 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.849732 19245 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.880935 16433 raft_consensus.cc:1203] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 20:12:47.950270 19195 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:47.950784 19195 tablet_bootstrap.cc:492] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:47.951117 19195 ts_tablet_manager.cc:1367] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.182s user 0.085s sys 0.042s
I20240501 20:12:47.951642 19249 ts_tablet_manager.cc:902] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Initiating tablet copy from peer 3b9f58142e4a4ccd9d0cbafc97d459c7 (127.28.136.69:32991)
I20240501 20:12:47.954046 19112 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:47.954521 19112 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Bootstrap complete.
I20240501 20:12:47.955029 19112 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.305s user 0.057s sys 0.010s
I20240501 20:12:47.955902 19112 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
I20240501 20:12:47.956344 19112 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: LEARNER
I20240501 20:12:47.956596 19112 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
W20240501 20:12:47.960101 13841 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.71:43621, user_credentials={real_user=slave}} blocked reactor thread for 114874us
I20240501 20:12:47.960180 19112 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:12:47.961194 14429 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:12:47.961623 14451 tablet_copy_service.cc:342] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Request end of tablet copy session 692bb14bda8046b9827f1091b0946088-c6213df9f24a4e63aec805f7dfacc75d received from {username='slave'} at 127.28.136.68:42323
I20240501 20:12:47.961756 14451 tablet_copy_service.cc:433] P 3b9f58142e4a4ccd9d0cbafc97d459c7: ending tablet copy session 692bb14bda8046b9827f1091b0946088-c6213df9f24a4e63aec805f7dfacc75d on tablet c6213df9f24a4e63aec805f7dfacc75d with peer 692bb14bda8046b9827f1091b0946088
I20240501 20:12:47.951999 19195 raft_consensus.cc:348] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:47.964109 19195 raft_consensus.cc:729] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:47.964387 19195 consensus_queue.cc:260] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:47.964987 19112 ts_tablet_manager.cc:902] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:47.967841 19195 ts_tablet_manager.cc:1398] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.017s user 0.003s sys 0.001s
I20240501 20:12:47.968641 14189 tablet_copy_service.cc:342] P 0e898d8ee23441908d1593c561b368ad: Request end of tablet copy session aeb70769b705493bb5196527311b142b-3808b45349de4959b929a99735e5db53 received from {username='slave'} at 127.28.136.72:35535
I20240501 20:12:47.968834 14189 tablet_copy_service.cc:433] P 0e898d8ee23441908d1593c561b368ad: ending tablet copy session aeb70769b705493bb5196527311b142b-3808b45349de4959b929a99735e5db53 on tablet 3808b45349de4959b929a99735e5db53 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:47.970785 14430 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:47.973731 19245 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.977701 16568 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
W20240501 20:12:47.987349 13973 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.74:45465, user_credentials={real_user=slave}} blocked reactor thread for 247154us
I20240501 20:12:47.987726 13764 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 10 to 11, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:47.992419 18983 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.993113 18983 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:12:47.993316 14714 tablet_copy_service.cc:140] P 3c0f925474374c18a6cd80cff94d4295: Received BeginTabletCopySession request for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 from peer c3224a319e2f4cb8a84bf9528209a9fb ({username='slave'} at 127.28.136.66:49721)
I20240501 20:12:47.993517 14714 tablet_copy_service.cc:161] P 3c0f925474374c18a6cd80cff94d4295: Beginning new tablet copy session on tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 from peer c3224a319e2f4cb8a84bf9528209a9fb at {username='slave'} at 127.28.136.66:49721: session id = c3224a319e2f4cb8a84bf9528209a9fb-4e21fb0e6bac46fbbd29bbd2f1bf43b4
I20240501 20:12:48.001776 19206 ts_tablet_manager.cc:1888] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:12:48.002151 19206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:48.004384 14714 tablet_copy_source_session.cc:205] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:48.013120 16433 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:48.016724 14298 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:48.022188 19195 ts_tablet_manager.cc:902] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: Initiating tablet copy from peer 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579)
I20240501 20:12:48.026707 19112 tablet_copy_client.cc:308] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:48.027496 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet 48770e5bebdf4d0b8766738665f7b560 from peer 692bb14bda8046b9827f1091b0946088 ({username='slave'} at 127.28.136.68:58785)
I20240501 20:12:48.027714 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet 48770e5bebdf4d0b8766738665f7b560 from peer 692bb14bda8046b9827f1091b0946088 at {username='slave'} at 127.28.136.68:58785: session id = 692bb14bda8046b9827f1091b0946088-48770e5bebdf4d0b8766738665f7b560
I20240501 20:12:48.030962 14583 tablet_copy_source_session.cc:205] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:48.033072 19112 ts_tablet_manager.cc:1875] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:48.032913 14694 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:48.038839 16433 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:48.038872 16435 raft_consensus.cc:1203] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 20:12:48.049185 19112 ts_tablet_manager.cc:1888] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:12:48.049511 19112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770e5bebdf4d0b8766738665f7b560. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:48.048866 14694 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:48.052862 19112 tablet_copy_client.cc:783] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:48.059620 19112 tablet_copy_client.cc:647] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:48.059733 19196 ts_tablet_manager.cc:1875] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:48.062261 19206 tablet_copy_client.cc:783] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:48.065013 19206 tablet_copy_client.cc:647] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:48.068632 19206 tablet_copy_client.cc:523] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:48.068804 19112 tablet_copy_client.cc:523] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:48.070817 19196 ts_tablet_manager.cc:1888] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 20:12:48.071254 19196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:48.072350 19112 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Bootstrap starting.
I20240501 20:12:48.077724 14167 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:48.084777 14429 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a5ca96c8a9a74413bff544306af10586 (127.28.136.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:48.090175 19206 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:48.098945 19196 tablet_copy_client.cc:783] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:48.099298 19196 tablet_copy_client.cc:647] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:48.099606 19195 tablet_copy_client.cc:308] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.67:38579
I20240501 20:12:48.107113 14189 tablet_copy_service.cc:140] P 0e898d8ee23441908d1593c561b368ad: Received BeginTabletCopySession request for tablet 39eb086c2b80482a98ff2baae96b9f60 from peer aeb70769b705493bb5196527311b142b ({username='slave'} at 127.28.136.72:35535)
I20240501 20:12:48.107306 14189 tablet_copy_service.cc:161] P 0e898d8ee23441908d1593c561b368ad: Beginning new tablet copy session on tablet 39eb086c2b80482a98ff2baae96b9f60 from peer aeb70769b705493bb5196527311b142b at {username='slave'} at 127.28.136.72:35535: session id = aeb70769b705493bb5196527311b142b-39eb086c2b80482a98ff2baae96b9f60
I20240501 20:12:48.110553 14189 tablet_copy_source_session.cc:205] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:48.112277 19195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39eb086c2b80482a98ff2baae96b9f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:48.119527 19196 tablet_copy_client.cc:523] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:48.120244 19195 tablet_copy_client.cc:783] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:48.120599 19195 tablet_copy_client.cc:647] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:48.122829 19195 tablet_copy_client.cc:523] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:48.127700 19195 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: Bootstrap starting.
I20240501 20:12:48.131031 16302 raft_consensus.cc:1203] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:48.146703 19196 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap starting.
I20240501 20:12:48.168676 19290 ts_tablet_manager.cc:902] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:48.170718 19249 tablet_copy_client.cc:308] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.69:32991
I20240501 20:12:48.171790 14451 tablet_copy_service.cc:140] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Received BeginTabletCopySession request for tablet c6213df9f24a4e63aec805f7dfacc75d from peer 0e898d8ee23441908d1593c561b368ad ({username='slave'} at 127.28.136.67:60763)
I20240501 20:12:48.171980 14451 tablet_copy_service.cc:161] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Beginning new tablet copy session on tablet c6213df9f24a4e63aec805f7dfacc75d from peer 0e898d8ee23441908d1593c561b368ad at {username='slave'} at 127.28.136.67:60763: session id = 0e898d8ee23441908d1593c561b368ad-c6213df9f24a4e63aec805f7dfacc75d
I20240501 20:12:48.174937 14451 tablet_copy_source_session.cc:205] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:48.176880 19249 ts_tablet_manager.cc:1875] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:48.204432 19231 raft_consensus.cc:1050] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:48.205924 19231 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:48.206740 19290 tablet_copy_client.cc:308] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:48.207705 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet eeaff69149ee4707beba512799b0d0df from peer 0e898d8ee23441908d1593c561b368ad ({username='slave'} at 127.28.136.67:47505)
I20240501 20:12:48.207866 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet eeaff69149ee4707beba512799b0d0df from peer 0e898d8ee23441908d1593c561b368ad at {username='slave'} at 127.28.136.67:47505: session id = 0e898d8ee23441908d1593c561b368ad-eeaff69149ee4707beba512799b0d0df
I20240501 20:12:48.211058 14583 tablet_copy_source_session.cc:205] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:48.212538 19290 ts_tablet_manager.cc:1875] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:48.214999 19195 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:48.215471 19195 tablet_bootstrap.cc:492] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: Bootstrap complete.
I20240501 20:12:48.215771 19195 ts_tablet_manager.cc:1367] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: Time spent bootstrapping tablet: real 0.088s user 0.053s sys 0.002s
I20240501 20:12:48.216607 19195 raft_consensus.cc:348] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:48.216957 19195 raft_consensus.cc:729] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aeb70769b705493bb5196527311b142b, State: Initialized, Role: LEARNER
I20240501 20:12:48.217223 19195 consensus_queue.cc:260] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: NON_VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: true } }
I20240501 20:12:48.229959 16435 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.230723 19229 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.238118 19249 ts_tablet_manager.cc:1888] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 20:12:48.238502 19249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6213df9f24a4e63aec805f7dfacc75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:48.238538 19195 ts_tablet_manager.cc:1398] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20240501 20:12:48.240116 14189 tablet_copy_service.cc:342] P 0e898d8ee23441908d1593c561b368ad: Request end of tablet copy session aeb70769b705493bb5196527311b142b-39eb086c2b80482a98ff2baae96b9f60 received from {username='slave'} at 127.28.136.72:35535
I20240501 20:12:48.240290 14189 tablet_copy_service.cc:433] P 0e898d8ee23441908d1593c561b368ad: ending tablet copy session aeb70769b705493bb5196527311b142b-39eb086c2b80482a98ff2baae96b9f60 on tablet 39eb086c2b80482a98ff2baae96b9f60 with peer aeb70769b705493bb5196527311b142b
I20240501 20:12:48.241614 13902 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.242408 19229 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.246543 19290 ts_tablet_manager.cc:1888] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 20:12:48.246881 19290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaff69149ee4707beba512799b0d0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:48.255153 16567 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.256332 14036 raft_consensus.cc:1203] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 20:12:48.256877 19235 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.268385 19249 tablet_copy_client.cc:783] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:48.268785 19249 tablet_copy_client.cc:647] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:48.276021 14694 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.276904 19229 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.277421 19249 tablet_copy_client.cc:523] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:48.283417 19249 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:48.284806 19290 tablet_copy_client.cc:783] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:48.285060 19290 tablet_copy_client.cc:647] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:48.288170 19290 tablet_copy_client.cc:523] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:48.289074 16302 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.289764 19229 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.293785 19305 ts_tablet_manager.cc:902] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Initiating tablet copy from peer c12d6cddb7f242b6af650a10bbc8eb69 (127.28.136.65:42939)
I20240501 20:12:48.302559 14298 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.303280 19235 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.306650 19290 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Bootstrap starting.
I20240501 20:12:48.315213 14563 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.315943 19229 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.336282 19229 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.342986 16568 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.349870 13757 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.357540 14563 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.362177 14694 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.371315 16302 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.377990 16433 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.390640 19305 tablet_copy_client.cc:308] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.65:42939
I20240501 20:12:48.391842 13926 tablet_copy_service.cc:140] P c12d6cddb7f242b6af650a10bbc8eb69: Received BeginTabletCopySession request for tablet 1840c628fc114b309da99d3c8a91656a from peer a5ca96c8a9a74413bff544306af10586 ({username='slave'} at 127.28.136.70:43029)
I20240501 20:12:48.392017 13926 tablet_copy_service.cc:161] P c12d6cddb7f242b6af650a10bbc8eb69: Beginning new tablet copy session on tablet 1840c628fc114b309da99d3c8a91656a from peer a5ca96c8a9a74413bff544306af10586 at {username='slave'} at 127.28.136.70:43029: session id = a5ca96c8a9a74413bff544306af10586-1840c628fc114b309da99d3c8a91656a
I20240501 20:12:48.393751 13905 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.395128 13926 tablet_copy_source_session.cc:205] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:48.397274 19305 ts_tablet_manager.cc:1875] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:48.425303 19305 ts_tablet_manager.cc:1888] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:12:48.425853 19305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1840c628fc114b309da99d3c8a91656a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:48.430859 14298 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.431824 14036 consensus_queue.cc:237] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:48.434708 14296 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:48.437309 14694 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.438289 19235 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.440631 16435 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.442710 19238 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.444006 13905 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.444825 16302 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.445504 19238 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.446293 19238 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.450999 19196 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:48.451573 19196 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Bootstrap complete.
I20240501 20:12:48.452090 19238 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.453805 19206 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:48.453997 19196 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent bootstrapping tablet: real 0.322s user 0.053s sys 0.007s
I20240501 20:12:48.454440 19206 tablet_bootstrap.cc:492] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Bootstrap complete.
I20240501 20:12:48.454897 14562 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.454978 19196 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:48.455324 19196 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c3224a319e2f4cb8a84bf9528209a9fb, State: Initialized, Role: LEARNER
I20240501 20:12:48.455660 19196 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:48.456048 19206 ts_tablet_manager.cc:1367] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.366s user 0.062s sys 0.004s
I20240501 20:12:48.457022 19206 raft_consensus.cc:348] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:48.457434 19206 raft_consensus.cc:729] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: LEARNER
I20240501 20:12:48.457257 19093 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240501 20:12:48.457723 19206 consensus_queue.cc:260] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: NON_VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: true } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:48.456377 19196 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:12:48.458112 13905 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.461143 13749 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8892d6dce05140babdeeb6c90c366b3b with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:48.461534 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session c3224a319e2f4cb8a84bf9528209a9fb-4e21fb0e6bac46fbbd29bbd2f1bf43b4 received from {username='slave'} at 127.28.136.66:49721
I20240501 20:12:48.461674 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session c3224a319e2f4cb8a84bf9528209a9fb-4e21fb0e6bac46fbbd29bbd2f1bf43b4 on tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with peer c3224a319e2f4cb8a84bf9528209a9fb
I20240501 20:12:48.462497 16433 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.463737 14562 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.465147 19206 ts_tablet_manager.cc:1398] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:12:48.466102 14714 tablet_copy_service.cc:342] P 3c0f925474374c18a6cd80cff94d4295: Request end of tablet copy session a5ca96c8a9a74413bff544306af10586-4e21fb0e6bac46fbbd29bbd2f1bf43b4 received from {username='slave'} at 127.28.136.70:40385
I20240501 20:12:48.466249 14714 tablet_copy_service.cc:433] P 3c0f925474374c18a6cd80cff94d4295: ending tablet copy session a5ca96c8a9a74413bff544306af10586-4e21fb0e6bac46fbbd29bbd2f1bf43b4 on tablet 4e21fb0e6bac46fbbd29bbd2f1bf43b4 with peer a5ca96c8a9a74413bff544306af10586
I20240501 20:12:48.468924 13759 catalog_manager.cc:5484] T 8892d6dce05140babdeeb6c90c366b3b P 1899220968384feebca3562bd7741a21 reported cstate change: config changed from index 12 to 13, VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) evicted. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.469928 14694 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.476372 16300 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.491051 14295 raft_consensus.cc:1261] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.493873 19238 consensus_queue.cc:1035] T 8892d6dce05140babdeeb6c90c366b3b P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.496042 14295 raft_consensus.cc:2932] T 8892d6dce05140babdeeb6c90c366b3b P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.500032 19105 raft_consensus.cc:1050] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER 1899220968384feebca3562bd7741a21 to VOTER
I20240501 20:12:48.501441 19105 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:48.502434 19112 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:48.502897 19112 tablet_bootstrap.cc:492] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Bootstrap complete.
I20240501 20:12:48.503260 19112 ts_tablet_manager.cc:1367] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Time spent bootstrapping tablet: real 0.431s user 0.049s sys 0.009s
I20240501 20:12:48.504371 19112 raft_consensus.cc:348] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
I20240501 20:12:48.504882 19112 raft_consensus.cc:729] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 692bb14bda8046b9827f1091b0946088, State: Initialized, Role: LEARNER
I20240501 20:12:48.505192 19112 consensus_queue.cc:260] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
I20240501 20:12:48.505910 19112 ts_tablet_manager.cc:1398] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:12:48.506675 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session 692bb14bda8046b9827f1091b0946088-48770e5bebdf4d0b8766738665f7b560 received from {username='slave'} at 127.28.136.68:58785
I20240501 20:12:48.506836 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session 692bb14bda8046b9827f1091b0946088-48770e5bebdf4d0b8766738665f7b560 on tablet 48770e5bebdf4d0b8766738665f7b560 with peer 692bb14bda8046b9827f1091b0946088
I20240501 20:12:48.512501 19290 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:48.513139 19290 tablet_bootstrap.cc:492] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Bootstrap complete.
I20240501 20:12:48.513527 19290 ts_tablet_manager.cc:1367] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.207s user 0.053s sys 0.006s
I20240501 20:12:48.514505 16435 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.514451 14563 consensus_queue.cc:237] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
I20240501 20:12:48.514561 19290 raft_consensus.cc:348] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } }
I20240501 20:12:48.514942 19290 raft_consensus.cc:729] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: LEARNER
I20240501 20:12:48.515245 19290 consensus_queue.cc:260] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } }
I20240501 20:12:48.516101 16300 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.516638 16567 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.516855 18705 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.517113 19249 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:48.517503 19305 tablet_copy_client.cc:783] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:48.517772 19249 tablet_bootstrap.cc:492] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Bootstrap complete.
I20240501 20:12:48.517814 19305 tablet_copy_client.cc:647] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:48.519595 19249 ts_tablet_manager.cc:1367] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Time spent bootstrapping tablet: real 0.236s user 0.060s sys 0.005s
I20240501 20:12:48.520826 19305 tablet_copy_client.cc:523] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:48.522244 19249 raft_consensus.cc:348] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } }
I20240501 20:12:48.522428 18705 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.522635 19249 raft_consensus.cc:729] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0e898d8ee23441908d1593c561b368ad, State: Initialized, Role: LEARNER
I20240501 20:12:48.522914 19249 consensus_queue.cc:260] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } }
I20240501 20:12:48.523778 19290 ts_tablet_manager.cc:1398] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:12:48.524628 14583 tablet_copy_service.cc:342] P a5ca96c8a9a74413bff544306af10586: Request end of tablet copy session 0e898d8ee23441908d1593c561b368ad-eeaff69149ee4707beba512799b0d0df received from {username='slave'} at 127.28.136.67:47505
I20240501 20:12:48.524816 14583 tablet_copy_service.cc:433] P a5ca96c8a9a74413bff544306af10586: ending tablet copy session 0e898d8ee23441908d1593c561b368ad-eeaff69149ee4707beba512799b0d0df on tablet eeaff69149ee4707beba512799b0d0df with peer 0e898d8ee23441908d1593c561b368ad
I20240501 20:12:48.526425 14430 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.526660 14560 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.527119 18958 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.527729 18958 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.529057 16302 raft_consensus.cc:1203] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240501 20:12:48.531869 19305 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Bootstrap starting.
I20240501 20:12:48.532413 19249 ts_tablet_manager.cc:1398] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:12:48.533247 14451 tablet_copy_service.cc:342] P 3b9f58142e4a4ccd9d0cbafc97d459c7: Request end of tablet copy session 0e898d8ee23441908d1593c561b368ad-c6213df9f24a4e63aec805f7dfacc75d received from {username='slave'} at 127.28.136.67:60763
I20240501 20:12:48.533427 14451 tablet_copy_service.cc:433] P 3b9f58142e4a4ccd9d0cbafc97d459c7: ending tablet copy session 0e898d8ee23441908d1593c561b368ad-c6213df9f24a4e63aec805f7dfacc75d on tablet c6213df9f24a4e63aec805f7dfacc75d with peer 0e898d8ee23441908d1593c561b368ad
I20240501 20:12:48.540117 13905 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.540203 14036 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:48.540436 18705 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.541246 19340 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.545475 18705 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:48.581825 16549 tablet_service.cc:1514] Processing DeleteTablet for tablet 8892d6dce05140babdeeb6c90c366b3b with delete_type TABLET_DATA_TOMBSTONED (TS 2a31af84f57c4ef18f0d98ff044229b6 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:45950
I20240501 20:12:48.596330 14036 raft_consensus.cc:1203] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 20:12:48.603549 19345 tablet_replica.cc:330] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: stopping tablet replica
I20240501 20:12:48.604461 19345 raft_consensus.cc:2229] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:48.605055 19345 raft_consensus.cc:2258] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:48.606866 19345 ts_tablet_manager.cc:1875] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:12:48.609529 14628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:36397, user_credentials={real_user=slave}} blocked reactor thread for 82429.9us
I20240501 20:12:48.618836 19345 ts_tablet_manager.cc:1888] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 20:12:48.619243 19345 log.cc:1196] T 8892d6dce05140babdeeb6c90c366b3b P 2a31af84f57c4ef18f0d98ff044229b6: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-8/wal/wals/8892d6dce05140babdeeb6c90c366b3b
I20240501 20:12:48.620399 13752 catalog_manager.cc:4830] TS 2a31af84f57c4ef18f0d98ff044229b6 (127.28.136.73:39773): tablet 8892d6dce05140babdeeb6c90c366b3b (table rebalance_test_table_2 [id=19a9545d59784fa196f5d5f54bfcc83e]) successfully deleted
I20240501 20:12:48.620889 18958 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.626024 19305 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:12:48.626602 19305 tablet_bootstrap.cc:492] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Bootstrap complete.
I20240501 20:12:48.626971 19305 ts_tablet_manager.cc:1367] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Time spent bootstrapping tablet: real 0.095s user 0.061s sys 0.002s
I20240501 20:12:48.626940 14429 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.627967 19305 raft_consensus.cc:348] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:48.628345 19305 raft_consensus.cc:729] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a5ca96c8a9a74413bff544306af10586, State: Initialized, Role: LEARNER
I20240501 20:12:48.628089 16433 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.628612 19305 consensus_queue.cc:260] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:48.630363 19305 ts_tablet_manager.cc:1398] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:12:48.631268 13926 tablet_copy_service.cc:342] P c12d6cddb7f242b6af650a10bbc8eb69: Request end of tablet copy session a5ca96c8a9a74413bff544306af10586-1840c628fc114b309da99d3c8a91656a received from {username='slave'} at 127.28.136.70:43029
I20240501 20:12:48.631418 13926 tablet_copy_service.cc:433] P c12d6cddb7f242b6af650a10bbc8eb69: ending tablet copy session a5ca96c8a9a74413bff544306af10586-1840c628fc114b309da99d3c8a91656a on tablet 1840c628fc114b309da99d3c8a91656a with peer a5ca96c8a9a74413bff544306af10586
I20240501 20:12:48.634455 16568 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.635315 14036 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.637142 13905 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.643590 13759 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.644205 13902 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:48.644850 19242 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:48.646095 14167 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:48.646965 19242 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 20:12:48.648983 14496 consensus_peers.cc:485] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 -> Peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Couldn't send request to peer 692bb14bda8046b9827f1091b0946088. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:48.659009 16568 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:48.659674 19242 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:48.663120 14694 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:48.664508 19242 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240501 20:12:48.674345 16300 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.676676 14562 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 1899220968384feebca3562bd7741a21 (127.28.136.74) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.684748 19230 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:48.690325 14694 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:48.703346 14036 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:48.704209 19186 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:48.705962 16567 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:48.709550 14429 raft_consensus.cc:1261] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:48.710559 14167 raft_consensus.cc:1203] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 20:12:48.712637 19186 consensus_queue.cc:1035] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:48.714607 14036 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:48.716159 13759 catalog_manager.cc:5484] T 59fdeaff3d92460c8ca59eee1144b127 P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:48.722999 14167 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:48.730291 13902 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:48.740293 14694 consensus_queue.cc:237] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } }
I20240501 20:12:48.741137 19130 raft_consensus.cc:1050] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:48.742367 19130 consensus_queue.cc:237] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:48.744931 16568 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.745682 19338 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.748982 13905 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.749636 19338 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.754509 14036 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.756959 14430 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.757107 16433 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.757248 14295 raft_consensus.cc:1203] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 20:12:48.758280 18958 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.758673 19105 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.762315 19105 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.765012 18958 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.767894 14563 raft_consensus.cc:1261] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.772421 16432 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.773203 14429 raft_consensus.cc:2932] T 59fdeaff3d92460c8ca59eee1144b127 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:48.777930 14036 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.781836 19105 consensus_queue.cc:1035] T af0f6a4c2e9c43509502a242b596d9da P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.783823 13759 catalog_manager.cc:5484] T af0f6a4c2e9c43509502a242b596d9da P 1899220968384feebca3562bd7741a21 reported cstate change: config changed from index 12 to 13, VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) evicted. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.784389 14563 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.788437 14430 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.792498 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af0f6a4c2e9c43509502a242b596d9da with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:48.793192 16568 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.800555 13905 raft_consensus.cc:2932] T af0f6a4c2e9c43509502a242b596d9da P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER aeb70769b705493bb5196527311b142b (127.28.136.72) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } }
I20240501 20:12:48.802589 19306 raft_consensus.cc:1050] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088: attempting to promote NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb to VOTER
I20240501 20:12:48.803984 19306 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } }
I20240501 20:12:48.839720 14563 raft_consensus.cc:1203] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 20:12:48.854233 14694 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:48.854224 16432 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:48.854983 19137 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:48.855504 13902 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:48.855731 19137 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:48.862906 19139 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:48.872897 14167 raft_consensus.cc:1203] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 20:12:48.875303 14563 raft_consensus.cc:1203] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 20:12:48.902431 14036 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 LEARNER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:12:48.903509 19212 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.920123 16300 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:12:48.920202 16432 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:12:48.920941 19212 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.921705 19212 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.954653 19173 raft_consensus.cc:1050] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: attempting to promote NON_VOTER 692bb14bda8046b9827f1091b0946088 to VOTER
I20240501 20:12:48.956207 19173 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } }
I20240501 20:12:48.969851 14694 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:12:48.970877 19393 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.973726 14167 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240501 20:12:48.974510 14236 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.72:39739, user_credentials={real_user=slave}} blocked reactor thread for 53635.8us
I20240501 20:12:48.974566 19051 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.976649 14169 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:12:48.977545 14563 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:12:48.982820 13905 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:12:48.983981 14295 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.984880 19051 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.988842 19395 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.989297 16433 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.990062 16568 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:48.990728 19051 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.991904 14036 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:48.991957 16302 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:48.992585 19137 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:48.993341 19137 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:48.994133 14295 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:48.995368 19212 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:48.997468 19137 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.007984 19408 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.009985 14560 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.010761 19051 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.016656 14694 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.018839 14429 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:49.020606 19130 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.022647 19051 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240501 20:12:49.026912 19306 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.040208 19051 raft_consensus.cc:1013] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: attempt to promote peer 0e898d8ee23441908d1593c561b368ad: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:12:49.044035 19137 raft_consensus.cc:1050] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad: attempting to promote NON_VOTER aeb70769b705493bb5196527311b142b to VOTER
I20240501 20:12:49.044652 19139 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.045833 19105 raft_consensus.cc:1050] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER c3224a319e2f4cb8a84bf9528209a9fb to VOTER
I20240501 20:12:49.045562 19137 consensus_queue.cc:237] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:49.047123 19105 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } }
I20240501 20:12:49.049085 14295 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.051959 13902 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.052160 14693 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.057835 14429 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.059512 16284 tablet_service.cc:1514] Processing DeleteTablet for tablet af0f6a4c2e9c43509502a242b596d9da with delete_type TABLET_DATA_TOMBSTONED (TS aeb70769b705493bb5196527311b142b not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:54294
I20240501 20:12:49.061305 13764 catalog_manager.cc:5484] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.063936 19051 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.069594 14693 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.075811 19212 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.078292 14038 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.080456 16300 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.083246 16432 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.091009 13762 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 reported cstate change: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "692bb14bda8046b9827f1091b0946088" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.098639 14694 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.103982 14169 consensus_queue.cc:237] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } }
I20240501 20:12:49.109433 14691 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.109652 14429 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.110352 13902 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.110579 19130 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.111137 19139 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.111757 14038 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.112509 19130 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.113401 19130 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.111299 14560 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.117748 19130 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.117889 13764 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 reported cstate change: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.119150 14036 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.122916 13902 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.066977 16433 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.123059 16300 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.125659 13751 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3808b45349de4959b929a99735e5db53 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:49.134311 14038 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.134406 13905 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.141381 14037 consensus_queue.cc:237] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
I20240501 20:12:49.144323 13762 catalog_manager.cc:5484] T 3808b45349de4959b929a99735e5db53 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 9 to 10, VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) evicted. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.150533 16302 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b [term 1 LEARNER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:49.151185 19139 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.151957 16432 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:49.152720 19139 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.150178 14693 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:49.154686 19139 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.154887 14429 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:49.155716 19139 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.159946 14038 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:49.160905 19139 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.170549 19422 tablet_replica.cc:330] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: stopping tablet replica
I20240501 20:12:49.171618 19422 raft_consensus.cc:2229] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:49.176487 14295 consensus_queue.cc:237] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } }
W20240501 20:12:49.177080 13972 consensus_peers.cc:485] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb -> Peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Couldn't send request to peer 692bb14bda8046b9827f1091b0946088. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:12:49.177412 14367 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.73:39773, user_credentials={real_user=slave}} blocked reactor thread for 186870us
I20240501 20:12:49.179908 14694 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:49.180079 14298 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.180486 19238 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:49.181875 14691 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.184335 14694 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 20:12:49.185072 19212 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.190932 14430 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:49.190749 19139 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240501 20:12:49.191624 13904 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:49.192961 19231 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:49.194062 14298 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.195022 14169 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:49.196022 19229 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:49.196659 19229 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:49.207043 16433 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 20:12:49.209403 19393 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.213033 13902 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.167829 14167 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.228524 16567 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.229022 14563 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:49.229182 18705 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.232025 14561 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.232985 14559 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:49.236217 19238 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:49.237206 18958 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.238202 19385 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.238369 14034 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.239285 18958 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.247593 16568 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.248461 19385 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.249619 14166 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.250516 14035 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 20:12:49.250830 18705 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.251510 14431 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.254362 19435 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.256377 14429 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.258636 19353 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.259094 19306 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.264568 19230 raft_consensus.cc:1050] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER 0e898d8ee23441908d1593c561b368ad to VOTER
I20240501 20:12:49.264603 14038 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c3224a319e2f4cb8a84bf9528209a9fb [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.265830 19230 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } }
I20240501 20:12:49.271116 16566 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.272111 13902 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.271705 14692 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 3c0f925474374c18a6cd80cff94d4295 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.273586 19242 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.275781 14035 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.276490 19242 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.277087 19105 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.277428 13757 catalog_manager.cc:5484] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 reported cstate change: config changed from index 13 to 14, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New cstate: current_term: 2 leader_uuid: "692bb14bda8046b9827f1091b0946088" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:49.278559 14691 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.279127 19230 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240501 20:12:49.279829 14165 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.280460 19230 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.280930 14431 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.283913 14295 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.284006 14147 tablet_service.cc:1514] Processing DeleteTablet for tablet fe1592bebd6b4a858987847446f15a16 with delete_type TABLET_DATA_TOMBSTONED (TS 0e898d8ee23441908d1593c561b368ad not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:44418
I20240501 20:12:49.285694 18958 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20240501 20:12:49.281603 19230 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.006s
I20240501 20:12:49.288347 19238 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.292766 14562 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.301465 14034 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.302235 19408 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.305871 14691 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.306169 13757 catalog_manager.cc:5484] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New cstate: current_term: 1 leader_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:49.315603 16436 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P 1899220968384feebca3562bd7741a21 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.325912 16435 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.326853 19242 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.329608 14169 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.333037 13903 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.342087 16432 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.345191 14428 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.350075 18958 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.364495 14563 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 20:12:49.367198 19393 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.367038 14298 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.369724 19408 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.370160 14168 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.371593 13905 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.372427 19230 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.372251 16436 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.372997 14563 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.179164 16303 raft_consensus.cc:1261] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.192391 16302 raft_consensus.cc:1261] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer c3224a319e2f4cb8a84bf9528209a9fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:12:49.205024 16304 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 20:12:49.383083 19426 consensus_queue.cc:1035] T 3808b45349de4959b929a99735e5db53 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.384819 19238 consensus_queue.cc:1035] T 58e126e98f1349cdb3e532ff49d3e926 P c3224a319e2f4cb8a84bf9528209a9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:12:49.385098 16304 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.388280 13764 catalog_manager.cc:5484] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 reported cstate change: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.393573 13904 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:12:49.394277 19426 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:12:49.394483 14560 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P a5ca96c8a9a74413bff544306af10586 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.398131 14430 raft_consensus.cc:2932] T 3808b45349de4959b929a99735e5db53 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1899220968384feebca3562bd7741a21 (127.28.136.74) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.403681 19353 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:49.406580 13902 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:49.407454 19422 raft_consensus.cc:2258] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:49.408018 13906 raft_consensus.cc:1261] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Refusing update from remote peer 692bb14bda8046b9827f1091b0946088: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 20:12:49.409009 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe1592bebd6b4a858987847446f15a16 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:49.410609 19395 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.410754 19242 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.410804 16568 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:49.411638 16304 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.417198 16566 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.417459 14164 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.417716 14033 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.424072 13763 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:49.425341 16436 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.430819 14166 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:49.433956 13761 catalog_manager.cc:5484] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3c0f925474374c18a6cd80cff94d4295" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:49.439011 19408 raft_consensus.cc:1050] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7: attempting to promote NON_VOTER 0e898d8ee23441908d1593c561b368ad to VOTER
I20240501 20:12:49.443295 19472 ts_tablet_manager.cc:902] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: Initiating tablet copy from peer a5ca96c8a9a74413bff544306af10586 (127.28.136.70:36397)
I20240501 20:12:49.408596 14693 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.449584 19471 raft_consensus.cc:1050] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69: attempting to promote NON_VOTER a5ca96c8a9a74413bff544306af10586 to VOTER
I20240501 20:12:49.450644 19422 ts_tablet_manager.cc:1875] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:49.451009 19471 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } }
I20240501 20:12:49.450719 14692 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.458389 16416 tablet_service.cc:1514] Processing DeleteTablet for tablet 3808b45349de4959b929a99735e5db53 with delete_type TABLET_DATA_TOMBSTONED (TS 1899220968384feebca3562bd7741a21 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60712
I20240501 20:12:49.459049 19395 consensus_queue.cc:1035] T fe1592bebd6b4a858987847446f15a16 P 692bb14bda8046b9827f1091b0946088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.460153 19242 raft_consensus.cc:1050] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586: attempting to promote NON_VOTER 692bb14bda8046b9827f1091b0946088 to VOTER
I20240501 20:12:49.461560 19242 consensus_queue.cc:237] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } }
I20240501 20:12:49.461560 16304 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P aeb70769b705493bb5196527311b142b [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.473299 16436 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.474097 19230 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.477632 16303 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.478307 14691 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.479009 16434 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.479249 14563 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.479933 19363 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.481168 19363 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.484812 19473 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240501 20:12:49.485272 13906 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.486449 14168 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.487784 16568 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.488483 19247 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.488701 14298 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.489296 19247 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.493957 14298 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 LEARNER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.494619 19363 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.495519 19363 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.497864 13901 raft_consensus.cc:2932] T fe1592bebd6b4a858987847446f15a16 P c12d6cddb7f242b6af650a10bbc8eb69 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 0e898d8ee23441908d1593c561b368ad (127.28.136.67) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } }
I20240501 20:12:49.500283 16434 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.501740 14034 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.510211 19473 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.509728 14559 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.513278 13904 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aeb70769b705493bb5196527311b142b (127.28.136.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } }
I20240501 20:12:49.518699 14037 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.519631 13903 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: NON_VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: true } } }
I20240501 20:12:49.524196 19472 tablet_copy_client.cc:308] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: tablet copy: Beginning tablet copy session from remote peer at address 127.28.136.70:36397
I20240501 20:12:49.525051 14583 tablet_copy_service.cc:140] P a5ca96c8a9a74413bff544306af10586: Received BeginTabletCopySession request for tablet 59fdeaff3d92460c8ca59eee1144b127 from peer 692bb14bda8046b9827f1091b0946088 ({username='slave'} at 127.28.136.68:58785)
I20240501 20:12:49.525252 14583 tablet_copy_service.cc:161] P a5ca96c8a9a74413bff544306af10586: Beginning new tablet copy session on tablet 59fdeaff3d92460c8ca59eee1144b127 from peer 692bb14bda8046b9827f1091b0946088 at {username='slave'} at 127.28.136.68:58785: session id = 692bb14bda8046b9827f1091b0946088-59fdeaff3d92460c8ca59eee1144b127
I20240501 20:12:49.528421 14583 tablet_copy_source_session.cc:205] T 59fdeaff3d92460c8ca59eee1144b127 P a5ca96c8a9a74413bff544306af10586: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:12:49.530344 19457 tablet_replica.cc:330] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: stopping tablet replica
I20240501 20:12:49.530534 19472 ts_tablet_manager.cc:1875] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:12:49.531098 19457 raft_consensus.cc:2229] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:49.531793 19457 raft_consensus.cc:2258] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:12:49.517632 16302 raft_consensus.cc:2932] T 58e126e98f1349cdb3e532ff49d3e926 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.533444 19457 ts_tablet_manager.cc:1875] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:49.536000 19475 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.542580 19477 tablet_replica.cc:330] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: stopping tablet replica
I20240501 20:12:49.544190 19477 raft_consensus.cc:2229] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:12:49.544718 19477 raft_consensus.cc:2258] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 20:12:49.547124 14627 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.66:44849, user_credentials={real_user=slave}} blocked reactor thread for 81608.3us
I20240501 20:12:49.547817 14036 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.548772 19408 consensus_queue.cc:237] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } }
W20240501 20:12:49.549399 14629 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.67:38579, user_credentials={real_user=slave}} blocked reactor thread for 68438.6us
W20240501 20:12:49.549837 14628 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.70:36397, user_credentials={real_user=slave}} blocked reactor thread for 48877.1us
I20240501 20:12:49.552028 14691 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.552273 19363 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.553458 14296 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.553504 18958 raft_consensus.cc:1050] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295: attempting to promote NON_VOTER a5ca96c8a9a74413bff544306af10586 to VOTER
I20240501 20:12:49.554518 18958 consensus_queue.cc:237] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } }
I20240501 20:12:49.562153 19471 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.563445 14295 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:49.565173 14427 raft_consensus.cc:1261] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.565673 14424 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:12:49.566767 19411 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.013s
I20240501 20:12:49.567952 19473 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.570230 19230 consensus_queue.cc:1035] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.571780 14558 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.573882 14690 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.574699 14558 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.575383 19051 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.575881 19173 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.582444 16567 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.583709 19051 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.586930 19051 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.589694 16566 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.593686 14560 consensus_queue.cc:237] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } }
I20240501 20:12:49.596531 19472 ts_tablet_manager.cc:1888] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:12:49.596510 13763 catalog_manager.cc:5484] T c6213df9f24a4e63aec805f7dfacc75d P 2a31af84f57c4ef18f0d98ff044229b6 reported cstate change: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.539848 13905 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.601670 14033 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.602632 19173 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.603506 14561 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P a5ca96c8a9a74413bff544306af10586 [term 1 LEARNER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
W20240501 20:12:49.604364 19450 log.cc:924] Time spent T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20240501 20:12:49.604787 14033 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.606283 19473 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } } }
I20240501 20:12:49.613147 13901 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.614216 19425 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.614413 14165 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } } }
I20240501 20:12:49.614270 16567 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } } }
I20240501 20:12:49.615974 13901 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.620800 19105 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 20:12:49.625972 14630 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.136.73:39773, user_credentials={real_user=slave}} blocked reactor thread for 167524us
I20240501 20:12:49.626312 13759 catalog_manager.cc:5484] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:12:49.628082 16566 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.629041 19353 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.631036 19242 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } }
I20240501 20:12:49.637949 14427 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } } }
I20240501 20:12:49.638475 14035 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: NON_VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: true } } }
I20240501 20:12:49.641956 13904 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } }
I20240501 20:12:49.643340 14429 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } }
I20240501 20:12:49.646339 19422 ts_tablet_manager.cc:1888] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240501 20:12:49.646597 19422 log.cc:1196] T af0f6a4c2e9c43509502a242b596d9da P aeb70769b705493bb5196527311b142b: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-7/wal/wals/af0f6a4c2e9c43509502a242b596d9da
I20240501 20:12:49.646471 14037 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } }
I20240501 20:12:49.647677 13752 catalog_manager.cc:4830] TS aeb70769b705493bb5196527311b142b (127.28.136.72:39739): tablet af0f6a4c2e9c43509502a242b596d9da (table rebalance_test_table_1 [id=68cf18b48118420fb6b65fcc9759272c]) successfully deleted
I20240501 20:12:49.439198 14431 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.652704 19472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fdeaff3d92460c8ca59eee1144b127. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:12:49.612697 14296 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } } }
I20240501 20:12:49.653306 16304 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } }
I20240501 20:12:49.657601 19457 ts_tablet_manager.cc:1888] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240501 20:12:49.659307 13759 catalog_manager.cc:5484] T 48770e5bebdf4d0b8766738665f7b560 P a5ca96c8a9a74413bff544306af10586 reported cstate change: config changed from index 11 to 12, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:12:49.661414 14298 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } }
I20240501 20:12:49.590718 16433 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.667205 19424 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.669740 19477 ts_tablet_manager.cc:1875] T 3808b45349de4959b929a99735e5db53 P 1899220968384feebca3562bd7741a21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:12:49.671938 16566 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.672772 19363 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.673067 19457 log.cc:1196] T fe1592bebd6b4a858987847446f15a16 P 0e898d8ee23441908d1593c561b368ad: Deleting WAL directory at /tmp/dist-test-taskzuvgUY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714594119446098-29217-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe1592bebd6b4a858987847446f15a16
I20240501 20:12:49.673913 13903 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.674738 19363 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.664381 14169 raft_consensus.cc:1261] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 LEARNER]: Refusing update from remote peer 3b9f58142e4a4ccd9d0cbafc97d459c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.681645 14559 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } } }
I20240501 20:12:49.687965 19425 consensus_queue.cc:1035] T c6213df9f24a4e63aec805f7dfacc75d P 3b9f58142e4a4ccd9d0cbafc97d459c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.661401 14166 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.688791 19363 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.689906 14166 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.693460 19230 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.696415 14295 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.662503 16436 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.669873 16433 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.697516 14169 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.699988 13751 catalog_manager.cc:4830] TS 0e898d8ee23441908d1593c561b368ad (127.28.136.67:38579): tablet fe1592bebd6b4a858987847446f15a16 (table rebalance_test_table_6 [id=245f85a2be2a4f4cb12f121c605da126]) successfully deleted
I20240501 20:12:49.638252 16432 raft_consensus.cc:2932] T 48770e5bebdf4d0b8766738665f7b560 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 692bb14bda8046b9827f1091b0946088 (127.28.136.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } }
I20240501 20:12:49.700508 14165 consensus_queue.cc:237] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } }
I20240501 20:12:49.701545 14560 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P a5ca96c8a9a74413bff544306af10586 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.703639 19242 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:12:49.704993 19472 tablet_copy_client.cc:783] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: tablet copy: Starting download of 0 data blocks...
I20240501 20:12:49.709138 16434 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.710275 19472 tablet_copy_client.cc:647] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: tablet copy: Starting download of 1 WAL segments...
I20240501 20:12:49.711207 14692 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.711380 19512 ts_tablet_manager.cc:902] T 58e126e98f1349cdb3e532ff49d3e926 P 692bb14bda8046b9827f1091b0946088: Initiating tablet copy from peer c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66:44849)
I20240501 20:12:49.711642 13750 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eeaff69149ee4707beba512799b0d0df with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:12:49.712772 16436 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.713732 16566 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 2a31af84f57c4ef18f0d98ff044229b6 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.714255 19472 tablet_copy_client.cc:523] T 59fdeaff3d92460c8ca59eee1144b127 P 692bb14bda8046b9827f1091b0946088: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:12:49.718291 16434 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } } }
I20240501 20:12:49.719859 13904 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.723860 14296 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 692bb14bda8046b9827f1091b0946088 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.724479 14692 raft_consensus.cc:2932] T c6213df9f24a4e63aec805f7dfacc75d P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 0e898d8ee23441908d1593c561b368ad (127.28.136.67) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.729593 13901 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.733629 19498 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.737862 16303 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 20:12:49.738305 14103 consensus_peers.cc:485] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad -> Peer 692bb14bda8046b9827f1091b0946088 (127.28.136.68:38031): Couldn't send request to peer 692bb14bda8046b9827f1091b0946088. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:12:49.738711 19434 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.739120 14425 raft_consensus.cc:1261] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer a5ca96c8a9a74413bff544306af10586: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.739966 19363 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.741966 14425 raft_consensus.cc:2932] T eeaff69149ee4707beba512799b0d0df P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.745657 14429 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.745878 14037 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.746222 19434 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.746598 19428 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.746975 13903 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.747642 19428 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.748320 16435 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.748900 19428 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:12:49.752292 19385 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.753413 14693 raft_consensus.cc:1261] T 39eb086c2b80482a98ff2baae96b9f60 P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Refusing update from remote peer 0e898d8ee23441908d1593c561b368ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:12:49.758463 19434 consensus_queue.cc:1035] T 39eb086c2b80482a98ff2baae96b9f60 P 0e898d8ee23441908d1593c561b368ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240501 20:12:49.758430 16431 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P 1899220968384feebca3562bd7741a21 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.763737 19470 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.763316 14034 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.769505 19230 consensus_queue.cc:1035] T eeaff69149ee4707beba512799b0d0df P a5ca96c8a9a74413bff544306af10586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.771343 13756 catalog_manager.cc:5484] T eeaff69149ee4707beba512799b0d0df P c12d6cddb7f242b6af650a10bbc8eb69 reported cstate change: config changed from index 12 to 13, VOTER c3224a319e2f4cb8a84bf9528209a9fb (127.28.136.66) evicted. New cstate: current_term: 1 leader_uuid: "a5ca96c8a9a74413bff544306af10586" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { promote: false } } }
I20240501 20:12:49.772049 14169 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.775588 19105 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.776319 18705 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.777629 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet eeaff69149ee4707beba512799b0d0df with delete_type TABLET_DATA_TOMBSTONED (TS c3224a319e2f4cb8a84bf9528209a9fb not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:56540
I20240501 20:12:49.779218 14036 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P c3224a319e2f4cb8a84bf9528209a9fb [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.780160 16302 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P aeb70769b705493bb5196527311b142b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.782058 14429 raft_consensus.cc:1261] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3b9f58142e4a4ccd9d0cbafc97d459c7 [term 1 FOLLOWER]: Refusing update from remote peer 3c0f925474374c18a6cd80cff94d4295: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:12:49.786732 13757 catalog_manager.cc:5484] T 39eb086c2b80482a98ff2baae96b9f60 P c3224a319e2f4cb8a84bf9528209a9fb reported cstate change: config changed from index 9 to 10, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New cstate: current_term: 1 leader_uuid: "0e898d8ee23441908d1593c561b368ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.787611 13904 consensus_queue.cc:237] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } }
I20240501 20:12:49.788120 18958 consensus_queue.cc:1035] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:12:49.792320 14165 raft_consensus.cc:1261] T 1840c628fc114b309da99d3c8a91656a P 0e898d8ee23441908d1593c561b368ad [term 1 FOLLOWER]: Refusing update from remote peer c12d6cddb7f242b6af650a10bbc8eb69: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:12:49.791747 19501 raft_consensus.cc:2932] T 4e21fb0e6bac46fbbd29bbd2f1bf43b4 P 3c0f925474374c18a6cd80cff94d4295 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } attrs { replace: true } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6" member_type: VOTER last_known_addr { host: "127.28.136.73" port: 39773 } attrs { promote: false } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } attrs { promote: false } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { promote: false } } }
I20240501 20:12:49.792001 13901 raft_consensus.cc:2932] T 39eb086c2b80482a98ff2baae96b9f60 P c12d6cddb7f242b6af650a10bbc8eb69 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 692bb14bda8046b9827f1091b0946088 (127.28.136.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "c3224a319e2f4cb8a84bf9528209a9fb" member_type: VOTER last_known_addr { host: "127.28.136.66" port: 44849 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "a5ca96c8a9a74413bff544306af10586" member_type: VOTER last_known_addr { host: "127.28.136.70" port: 36397 } attrs { replace: true } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false replace: true } } peers { permanent_uuid: "aeb70769b705493bb5196527311b142b" member_type: VOTER last_known_addr { host: "127.28.136.72" port: 39739 } attrs { promote: false } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: NON_VOTER last_known_addr { host: "127.28.136.68" port: 38031 } attrs { promote: true } } }
I20240501 20:12:49.792937 19471 consensus_queue.cc:1035] T 1840c628fc114b309da99d3c8a91656a P c12d6cddb7f242b6af650a10bbc8eb69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:12:49.748121 14694 raft_consensus.cc:2932] T 1840c628fc114b309da99d3c8a91656a P 3c0f925474374c18a6cd80cff94d4295 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a5ca96c8a9a74413bff544306af10586 (127.28.136.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "3b9f58142e4a4ccd9d0cbafc97d459c7" member_type: VOTER last_known_addr { host: "127.28.136.69" port: 32991 } } peers { permanent_uuid: "692bb14bda8046b9827f1091b0946088" member_type: VOTER last_known_addr { host: "127.28.136.68" port: 38031 } } peers { permanent_uuid: "3c0f925474374c18a6cd80cff94d4295" member_type: VOTER last_known_addr { host: "127.28.136.71" port: 43621 } attrs { replace: true } } peers { permanent_uuid: "c12d6cddb7f242b6af650a10bbc8eb69" member_type: VOTER last_known_addr { host: "127.28.136.65" port: 42939 } } peers { permanent_uuid: "0e898d8ee23441908d1593c561b368ad" member_type: VOTER last_known_addr { host: "127.28.136.67" port: 38579 } } peers { permanent_uuid: "1899220968384feebca3562bd7741a21" member_type: VOTER last_known_addr { host: "127.28.136.74" port: 45465 } attrs { promote: false } } peers { permanent_uuid: "2a31af84f57c4ef18f0d98ff044229b6"